Developing Stock-Market Bar Software

Posted on Tue 19 February 2019 in Python

I want this blog post to reflect my thoughts, choices, design-decisions and learning-curve through what became two months of intense work and nearly 3.000 lines of code. I don't necessarily want it to be very technical, I want it to reflect what decisions I made as an amateur programmer trough this project.

What have I gotten myself into...

I recently got the job as a board member of a voluntary student organization at my university. The organization is called FinanceLab and it's activities revolves around connecting students at the faculty for Economics and Management with top-tier financial companies and organizations.

Due to the very particular interest of our members, FinanceLab has every year held a stock-market bar at the university. A grand event with more than 1200 participants each year! The concept of stock-market bar is really rather simple: When purchaces are made, the price of the alcoholic drinks rise, when nothing is being sold the prices drop.

FinanceLab had, for a few years, been managing this price-action through software of which they had no or little control. They also wanted to add features and graphical flair. Enter my boldness and ignorance...

I, as Alice, went down the rabbit hole bringing nothing but a bit of bravery a fair bit of arrogance and no idea about the aptitude required of what I was heading towards.

So, how do I actually go about this?

I didn't approach this issue completely without any previous knowledge. I have been programming on a hobby basis almost all my adult life and thankfully have access to any programmers sharpest weapon: Stack Overflow ;)

Jokes aside, I needed some way of approaching this that would help me structure the project and make sure I developed all the tools I needed in the right order and using the right modules.

At the first glance, I figured I would need:

  • Some way of inputting data (preferably over the internet/LAN)
  • A visual interface visible for everyone attending the party
  • A backend to do all price-calculations (and figure out the maths behind this)

Now I at least had some kind of a plan.

Fix it with intelligent design or brute-force it in hackathons till it works?

I haven't ever been very interested in graphical design. So I thought I should rather fix the backend things first and then deal with all the visuals and flair later.

Guess I'll just start somewhere...

Obstacle #1 - How do I get user input?

I hadn't ever really written much networking capable code so I had a lot to learn! My interest for Linux and nerdy computer-stuff in general fortunately saved a lot of painful debugging hours with regard to understanding networking etc. The way I saw it, I had two options. Make an application and use web-sockets or build a small website to handle all traffic.

After messing around with the Python Sockets module for a bit, I figured - there must be a 'higher-level' way of during this. Also, I wanted my input-server to be as system-agnostic as posible. I didn't want to maintain all different kinds of operating systems etc. - a website would be perfect!

Enter Flask:

Flask is an amazingly well built framework for creating small (but yet capable) website only using Python (well... among A LOT of other languages etc. but the developer can almost get through exclusively seeing the Python code.)

The other really neat thing about Flask is it's really great integration with SQLAlchemy , which I also first started using during this project. SQLAlchemy is a database-agnostic module for interacting with databases. It made writing queries really easy and also switching between my dev-environment's SQLite database to the actual MySql database completely flawless.

This meant that I was able to in a week or so, get a neat-looking website up and running with a database system, user-login, access-control, registration of sales etc. A few hundred lines of code and all this was working - I was happy!

Obstacle #2 - Well.. I guess I need to calculate som prices?

So I only really had some very superficial ideas about how the system should calculate the prices. I knew that I only had visibility of one side of the market - namely the buy-side. I needed to somehow have price-actions every time a sale happened and otherwise have a time-decay of sorts that would lower the price as time passed.

I also wanted to make sure that I wrote everything to simulate liquidity in the market - therefore I thought randomness would make it perform sort of like a stock-market and not be too linear or rigid.

I ended out designing the following: 

...
def price_action(self):
    def price_increase_factor(increase, sold):
        def factors():
            factor = normal(loc=0.0, scale=increase * 2)
            factor = sqrt(factor ** 2)  # factor > 0

            return round((factor / 100), 5)

        increases = 0.0

        for _ in range(int(sold)):
            increases += factors()

        if increases > 0.15:
            increases = 0.15

        return increases + 1

    def price_decrease_factor(secs_passed, decay, latest_price, min_price):
        sens = (secs_passed * log(latest_price / min_price)) / 100
        real = normal(loc=0.0, scale=sens + (decay / 2)) / 100

        if real > 0:
            real = real * 0.30

        return 1 + round(real, 3)

    self.sales['increase_factor'] = self.sales.apply(lambda x: price_increase_factor(x['increase_sensitivity'],
                                                                                     x['products_sold']), axis=1)

    self.sales['decrease_factor'] = self.sales.apply(lambda x: price_decrease_factor(x['latest_sale'],
                                                                                     x['decay_sensitivity'],
                                                                                     x['latest_price'],
                                                                                     x['minprice']), axis=1)

    def price_change(price, min_price, products_sold, increase_factor, decrease_factor):
        if products_sold == 0:
            price = price * decrease_factor
        else:
            price = price * increase_factor

        if price <= min_price:
            price = min_price

        if price > 110.0:
            price = 110.0

        return round(price, 1)

    self.sales['new_price'] = self.sales.apply(lambda x: price_change(x['latest_price'],
                                                                      x['minprice'],
                                                                      x['products_sold'],
                                                                      x['increase_factor'],
                                                                      x['decrease_factor']), axis=1)
    return self.sales
...

The above small method is really rather simple and even turned out to be surprisingly fast. It relies heavily on the Pandas and Numpy modules which I've worked with extensively and absolutely love.

Let me explain. The price_increase_factor() method only takes two arguments. The increase parameter, which is set in the control-panel I showed above (integer between 1 and 5) and the amount of x product sold. It then estimates a normal distribution with the parameters of an average of 0 and a standard deviation of the increase parameter times two. It then calculates the absolute value of the factor and returns it to the dataframe.

The price_decrease_factor() method is a tiny bit more complex, however not too bad. I wanted some way to weight the passed time, relationship between minimum-price and current price and a user-defined decay-factor. By writing a lot of tests and a lot of tedious plotting, I finally had a functioning method for providing a decay that would ensure a well functioning time-decay (the user-defined decay parameter is the same as the increase one) .

I used Pandas' apply function to map the values to a dataframe, which then returned the required values to generate a new price.

These methods were run every other second and weren't really too compute intensive.

I just (while writing this) realize that a very obvious optimization would be not to calculate both increase_factor and decrease_factor for all products, when only one is used...

Obstacle #3 - So far, so good. But crap - I need some visuals!

I needed a few things to get my visuals of the application to work, be functional and get really good looking:

  • A chart for each drink for sale (beer, drinks and a special long-island sort of drink).
  • A "Breaking News" tile with user-input news-articles.
  • A logo in the top with our organizations logo.
  • Financial trading tickers in the bottom of the screen.

My initial idea was, if I was able to host everything in a Flask-like webserver, it would be really portable and easy to use - also for the next couple of years. So everything i needed was a small VPS and a computer connected to the screen This would eliminate all needs of distributing across systems and kill dependency-hell.

My first idea was to use Plotly's latest invention Dash . I unfortunately had a lot of issues dealing with Dash and the fact that the documentation isn't very mature didn't help. Most of my issues was however related to live-plotting, which I know can be... kind of tough. I however really admire Dash as a product and idea - it really has a lot of amazing potential - maybe next year ;)

After a lot of tinkering and reading I eventually settled upon PySide2 . PySide2 is an implementation of the famous QT-framework in python. I was stoked when I stumbled upon this! Embedding Matplotlib charts in PySide2 was a breeze and the QtDesigner is just magnificent and provides for some really easy handling of most of the design/stylesheet stuff.

I think it ended up looking really well - kind of sad that we had to use white instead of grey for the labels etc. but that was to do with the projector.

There were a lot of animations on the screen that you, reader, unfortunetly can't see from the static image right now.

I even - because of our finance-nerdy tendencies - had the idea to implement technical indicators that popped up every 15 minutes with new data in candlestick (OHLC) format with Bollinger Bands and moving averages:

I don't think diving into the code of the visuals is very interesting - let's just say it was a lot of fiddling and nerve wracking debugging and lots and lots of lines of styling... and QTimer objects :D

I may update the blog later, if I think some aspects of the visuals/frontend are particularly interesting.

Obstacle #4 - It doesn't work... Who would have thought universities had secure networks?

So, everything was fine and dandy - I had a lot of preliminaries out of the way and was sort of ready to get going. So I fired up the Flask server using the mod_wsgi Apache server-module aaaand... I couldn't connect from my phone to computer while on the university network. I try to look at the output of ifconfig and I could tell, they were on different subnets?! FUDGE!

My initial idea was just to host the Flask server locally and not having to deal with the latency of and costs of running a VPS. I was thinking a lot about this issue but kept on ending up at a dead end. I therefore contacted my friend (a Computer Science student), Ulrik Boll Djurtoft and asked him, if there was some way of hacking around this issue. He told me that he had an available server, which could host the system and we thought the best way to solve it was to host Flask + MySql server at the server and have my machine remotely connect to this during the event.

Suddenly there was a lot that could go wrong - speaking about noisy WiFi networks at the event, latency etc. We, however, got a really sketchy idea... What if, I connected trough a Zerotier connection from my machine to my friend's server. Have his server forward all traffic to port 80 incoming to a sub domain name through the tunnel to my computer on the university network!

I was baffled! I didn't even know this sort of thing was possible. It ran extraordinarily well! It was completely agnostic of the network and allowed for a lot more flexibility - also we didn't have to deal with tricky DHCP servers and my machine suddenly changing IP etc.

I even, for a proof of concept tried connecting trough a VPN from San Paolo, Brazil to the server - this was a cool solution!

This is the kind of hacking-creativity which just really makes me so intriged about IT. It perhaps wasn't the most elegant solution but it was such a great execution!

When the party is over...

Although I, by no means, am a professional programmer it did work. And it worked really well. No dropped packets no unregistered sales, all price-operations worked and the QT framework didn't make circles around itself. However I am certain that there are smarter ways of doing a lot of the code/infrastructure stuff. However, I'm kind of proud of my end result.

I love coding. That was the reason I even considered during this project. I had no financial gain, no help outside my above mentioned friend (well besides tremendous support from friends and collegeaus at FinanceLab) . I've ended up with maybe a pointy star on the CV and some good old street credit at my school. But it really boils down to during something you love and are passionate about. Showing the world your skills and sometimes taking on a task that is 100% too large, because trust me, you'll learn!

What I learned trough this really fun and rather challenging experience:

  • Git and version control can save your life.
  • Coffee solves problems.
  • Managing time and drawing your thoughts/design ideas is incredibly helpful.


Special thank to all the hardworking and amazingly intelligent people, whom have created and contributed to amazing open-source libraries that made this entire thing posible. Also to all the random strangers on Stack Overflow/Github whose code I've Frankensteined into this project. If all the work was for nothing, then you did at least make sure a lot of really drunk university students had an amazingly fun and interesting party.

These projects include:

  • Python
  • Numpy
  • Pandas
  • PySide2
  • Matplotlib
  • Flask
  • Zerotier
  • SQLAlchemy
  • Internet strangers and many more python-modules

pandas data algorithms full-stack development databases QT