Project pools

A little more than half a year into the public existence of Horizon EDA people noted that it’s far from optimal that that projects require all used items to be present in the pool. Apart from complicating sharing projects this meant that an updated item in the pool would automatically propagate into the project without any way to review the changes.

To fix this, I introduced the pool cache that keeps a copy of pool items used in projects. Even though it stood the test of time pretty well with a minimum of changes required, it had some shortcomings. To my surprise, no one brought up any of these.

  • The cache is mostly write-only, once an item is in there, there’s no easy way to apply one-off modifications
  • The pool browsers for selecting parts and the like aren’t aware of the cache and show items as they’re in the pool, not as they’re cached
  • Items only present in the cache and missing from the pool don’t show up in the pool browsers at all, making it difficult to use them
  • Just selecting items in pool browsers in the board or schematic editor adds them to the cache
  • 3D models aren’t found in some circumstances
  • There’s no convenient way to add pool items only relevant to one particular project such as pads that connect to bare glass LCD panels (oddly specific, I know)

Apart from that I had plans to make it easier to use items from multiple pools in a project.

Quickly, I came to the conclusion that the best way to solve all of these issues in one go is to replace the pool cache by project-specific pools. This means that every project is now accompanied by a pool that includes one or more other pools. While pools supported including other pools since quite some time, it’s been a rather niche feature that probably didn’t see that much use. As this feature has now become essential to projects, some rough edges had to be smoothed out, such as supporting recursive inclusions and enabling incremental pool updates for included pool items.

Using a pool item in a project will automatically copy it from its original pool into the project pool so it’s kept with the project, similar to how the cache worked.

Multiple pools

Over the course of time the main pool accumulated tens of thousands of generated capacitors and resistors of various manufacturers. Even though there have been improvements in Horizon EDA to keep performance at acceptable levels with a high number of parts, this trajectory isn’t exactly sustainable. Also people might opt to use workflows that don’t require exact MPNs for passives.

To solve both of these issues, each series of generated passives got moved into their own pool, so users can select exactly the pools needed for each project and no single pool will grow too large in size.

With there now being multiple pool, there comes the difficulty of keeping all of them up to date. Instead of having to open each pool individually to check for updates in the remote tab, there now is the “Pools” window that provides a one-stop shopping experience for keeping all pools updated:

In a similar spirit, the pools window now also takes care of providing a list of all pools available for download:

Pools are organized in 3 levels:

  • Core: The main pool
  • Extra: Pools maintained by the Horizon EDA project that augment the main pool
  • Community: Pools maintained by members of the community

A list of all pools is maintained in the horizon-pool-index repo. An Action then compiles the pools added in this repo into a single easy-to-parse JSON available at pool-index.horizon-eda.org/index.json.

If you happen to manage a pool of obscure parts that don’t really fit into the main pool, but still want to share it with other people, feel free to open a pull request.

Release?

The introduction of project pools was quite a big change and I didn’t feel quite comfortable with doing a stable release only about two months after this feature initially landed and it required some further clean-up work to make best use of the new concepts.

Version 1.5.0 is scheduled to be released around end of May.

Website stats

A while back I got aware of counter.dev offering free and privacy-friendly web analytics. Since I’ve been curious how many people find their way to this blog or the landing page, I decided to add it to both pages.

horizon-eda.org averages about 90 visits per week, with around 40% finding their way there via Google. I was surprised to see that the majority (26%) of the visitors is from Germany and the United States being second place at around 20%. With there living four times as many people in the states than in Germany, it seems like Horizon EDA (or at least the landing page) is significantly more popular in Germany than in other countries. Apart from a rather lengthy, but now inactive thread on the German mikrocontroller.net forum and some talks at the GPN, all of the content related to Horizon EDA is in english, so it’s probably not only for the language barrier.

About half of the visits originate from Linux (probably including Android) and 40% come from computers running Windows. Macs account for less than 10% of all visits, so not that many people will be disappointed to find out hat Horizon EDA doesn’t run on mac OS. Around three quarters of all visits happen from regular computes, i.e. not a mobile device.

The stats for this blog look roughly the same with the difference that the blog only averages around 20 visits a week. Linking to it in a submission about the Hershey fonts on Hacker News temporarily doubled that figure.

Text editor in properties

In the course of adding support for multi-line text, the text entry in the properties panel grew into a text area to facilitate entering more than one line. This however made editing single-line texts harder since pressing the enter key inserted a line break rather than committing the edit. Getting annoyed by this, the text property editor now uses the smart text editor from the “Enter datum” tool that’s either a single-line entry or a multi-line text area depending on the text to be edited:

Digi-Key stock info provider

Picking from the thousands of passives in the pool has turned out to be next to impossible without having the stock information directly in the part browser. When kitspace’s partinfo API was having some hiccups, I decided to explore adding new providers to fill that column. Since I almost exclusively shop at Digi-Key and they offer a free API with a rather generous quota of 1000 queries per day, I went ahead implementing a second StockInfoProvider.

The first obstacle to overcome is that the Digi-Key API requires additional OAuth2 authentication of the user accessing the API apart from the usual API keys. The thing with OAuth2 authentication is that it requires a callback URI the user is redirected to after having authenticated and Horizon EDA not being a web app, so I had to improvise.

Since I didn’t want to run any server-side application, everything’s handled on the client side and each user has to obtain their own API key. Now the question is, how to get OAuth code from the web into a desktop app?

For that purpose, I set up a simple static page with minimal Javascript that displays the code from the URL parameters and provides a convenient way to copy it to the clipboard:

Once the user has pasted this code into Horizon EDA, the app uses it and the API keys to obtain the tokens to ultimately access the API.

To make best use of the API quota and avoid doing queries that the user isn’t interested in, the API is only queried after pressing the Load button in the status bar:

Overbar in texts

Similar to other schematic drawing tools, starting a pin name or any other text with a tilde will draw a bar on top of it to indicate inverted signals:

Precisely speaking, a tilde toggles the overbar, so writing Pull ~RESET~ low will be rendered as

Pull RESET low

Starting a new line implicitly turns of the overbar. Use a double tilde to output a literal tilde.

Set increment context menu

Numerical entries with up/down buttons are super annoying to use when the desired increment is smaller than the increment used by the buttons. With the buttons being now useless, one has to revert to awkwardly typing in numbers. As it’s not always possible for the application to anticipate the desired increment, all dimension entries now have a context menu for setting the increment:

Pictures in PDF export

Thinking of easy things to hack on that provide instant gratification, adding pictures to exported PDFs came to my mind. Less than 100 lines of code later, that idea became reality.

Experimenting with semi-transparent images I noticed that translucent areas appear darker in the PDF than intended. Taking a closer look revealed that these pictures are also rendered incorrectly by Horizon EDA itself. After poking around a bit more, I discovered that pictures are rendered correctly directly after they’re imported, but show up too dark after the schematic file is reopened. Here’s what should be a white to transparent gradient on top of red objects:

This made me suspicious about various parts of the stack disagreeing about the use of premultiplied alpha, i.e. storing any pixel color value multiplied by the value of the alpha channel.

Googling “gdk pixbuf premultiplied alpha” brought up this excellent article that had the answer to all my trouble:

Horizon EDA uses Gdk’s Pixbuf loader to convert imported pictures of any format into a ARGB32 buffer. This pixel data doesn’t use premultiplied alpha, as does the OpenGL renderer and the PDF export. So far so good. For persisting pictures to disk however, I went with cairo’s PNG import and export functions without realizing that cairo’s image surfaces do use premultiplied alpha. Due to this mismatch, image data loaded from disk that uses premultiplied alpha will be used by code that assumes non-premultiplied alpha, leading to wrong colours in semi-transparent areas.

The knee-jerk fix to this would be to convert the pixel data to premultiplied alpha before handing them over to cairo, but looking at the code revealed that cairo itself converts it premultiplied alpha pixel data to non-premultiplied in the process of writing the PNG. To avoid potentially lossy conversions, I decided to directly use libpng to persist picture data to disk, preserving non-premultiplied alpha.

With this fix in place, the test image is now rendered correctly:

Switching sheets with navigation buttons

Inspired by switching layers with the back/forward buttons on the mouse, it’s now possible to switch sheets using these buttons as well.

Copy/paste net labels

In most schematic entry tools, connectivity can be created by copying labels from one place to another. Due to Horizon’s netlist-first approach to schematics, this wasn’t possible until now since copying a label without a symbol attached didn’t preserve the net assigned to the label.

By including net information in the copied data, labels keep their net in the paste tool an can be connected to pins or junctions to create connectivity.

Move junctions connected to symbols

Moving a symbol now automatically moves junctions connected to it to prevent bent net lines:

Part flags

Some parts aren’t parts in the actual sense of being a thing that one has to order and place on the board, such as test points or pads for pogo pin connectors. To avoid confusion, it’s desirable that parts of this kind are omitted from the BOM and pick&place data.

With the addition of flags, it’s now possible to specify this at the part level:

Layer colors everywhere

To make it easier to recognize layers in the property panel and other places, the layer color is shown next to the layer’s name: