Dear Tumbleweed users and hackers,

Nothing is stopping Tumbleweed – it’s still full steam ahead with 7 snapshots released in one week, which means daily snapshots without interruptions—trying to get a new streak record? let’s see! So far we’re at 14 days of release without a gap. So far, the highest streak was 18 if I’m not mistaken (2021/1116-1203). In any case, these are just nice stats, but the quality of the snapshots has always been more important to us than the number of snapshots. And I’m convinced the Tumbleweed users to see this the same way.

The last 7 snapshots (0811.0817) delivered these changes:

• Linux kernel 5.19.1
• GNOME 42.4 (completed, incl. gnome-shell and gnome-desktop)
• KDE Frameworks 5.97.0
• hdf5 1.12.2
• PostgreSQL 14.5
• Mozilla Firefox 103.0.2
• binutils 2.39
• git 2.37.2
• libwacom 2.4.0: support new devices, like Lenovo 14s Yoga, Samsung Galaxy Book Pro 360, and more
• wxWidgets 3.2.0

The Staging projects are well used, and the forge is hot, pressing things like:

• Linux kernel 5.19.2
• KDE Gear 22.08.0
• Boost 1.80.0
• systemd 251.4
• glibc 2.36: meta bug tracking failures: https://bugzilla.opensuse.org/show_bug.cgi?id=1202207
• Shadow 4.12
• fmt 9.0

“Jeff Wayne’s Musical Version of The War of the Worlds” has been a turning point in my life in many ways. It was one of the first non-classical albums I listened to. It was the starting point in my ability to understand spoken English.

The first steps from classical

My parents only listen to classical music. Even Bartók is too modern for them. In my household growing up, I was only exposed to classical music. Yes, I heard some pop-music on the streets, but I was told that it’s just noise, not music. I must admit that even to this date I mostly agree with this statement :-)

However, today I do not listen only to classical music. I still recall the first album that I liked and was not fully classical. It was Hooked on Classics played by the Royal Philharmonic Orchestra. Tons of familiar classical melodies played in the style of pop music of that time. I listened to these albums countless times.

Once the damage was done, I started to listen other non-classical works. From the early years I recall the names of Richard Clayderman and Kitaro. Clearly not classical music any more, even some electronic instruments, but still very different from mainstream pop music.

Understanding spoken English

In high school one of my classmates lent me an album: Jeff Wayne’s Musical Version of The War of the Worlds. First I listened to it as I loved the music and the story on which album was built. Then I realized that it can help me to understand spoken English.

It was right after 1989, when Hungary changed to a democracy. The Russian troops were still in the country, but I was in the first high school year where learning Russian was not mandatory any more. My primary foreign language in high school was German, the secondary was English. There was a glut of Russian language teachers and barely enough teachers for other languages. We had one or two English lessons a week, and very minimal chance to listen to real English pronunciation. At that time there was no YouTube, etc. We had a satellite TV receiver, but I could not follow spoken English there at all, as in school I never heard anything close to real English…

When I first listened to The War of the Worlds, I could barely understand anything, even when I was reading the text from the album cover. After a while I realized that repeated listening and reading the book, things “clicked” and I started to understand the language better. Then I started listening to the album not just for the music but to check if my understanding of spoken English improves. After a while I could follow the narrator and the singers even without having the album cover at hand.

The good thing is that understanding spoken English did not stop at this album. It was as very important milestone. From that time on, I could pick up more English from the television. Of course high school level English provided just a very basic level of understanding, which I would later build on to greatly improve my English skills. That’s another story, not related to music…

Listen to the album on TIDAL: https://listen.tidal.com/album/2917051

Read my blog about Discogs to learn about my music collection: https://peter.czanik.hu/posts/discogs/

The month of August is hot for openSUSE Tumbleweed as snapshots appear to be rolling out daily.

The trend this week is like Tumbleweed on cruise control just rolling out snapshot after snapshot.

Among the updated packages in snapshot 20220816, postgresql14 14.5 made a splash with fixing a Common Vulnerability and Exposure; with CVE-2022-2625, the extensions use of CREATE OR REPLACE or CREATE IF NOT EXISTS are not being adhered to according to the documented rules and attacker can run arbitrary code as the victim role, which may be a superuser. PostgreSQL is blocking this attack in the core server, so there is no need to modify individual extension scripts. Moving on to a more lighter subject, the snapshot provided an update of filesystem utility xfsprogs 5.19.0. The newer version update provides more autoconf modernization and fixes a memory leak. It’s counterpart, xfsdump 3.1.10, fixed bind mount handling that was corrupting dumps and removed Data Management Application Programming Interface support. Xfce users can now have window capture in HiDPI mode thanks to an update of xfce4-screenshooter 1.9.11.

KDE Frameworks 5.97.0 glided into snapshot 20220815 and gave Plasma Desktop users several fixes. Frameworks updated blur and other window effects when the dialog changes size and the password storage KWallet Framework introduced a Secret Service API. User Interface framework Kirigami added workaround for the Qt horizontal scroll-view bug. KIO had an update to better prevent duplicate bookmarks for the same Hypertext REFerence. Text editor vim saw its second update of the week; its 9.0.0203 version had some fixes for invalid memory access and a fix for extra space of virtual text when ‘linebreak’ is set. The diagnostic, debugging and instructional userspace package strace updated to version 5.19. The update had changes in behavior and implemented some decoding socket option and netlink attributes. The last package to update in the snapshot was hdf5 1.12.2; this general purpose library and file format for storing scientific data dropped one patch, disabled another and enabled the rpm and deb CPack generators on Linux.

Snapshot 20220814 updated the distribution to Linux Kernel 5.19.1. Nearly a third of all the updates for the kernel were related to bluetooth and most of those were for the RTL8852C wireless module. An update of gnome-shell 42.4 improved the overview animation performance and had a fix for remembering the set up of bluetooth devices. GNOME’s layout and text rendering package pango updated to 1.50.9 and fixed a thread-safety problem. There was a minor update to the boot splash package plymouth; the update can be used to check the secure boot configuration and put a red warning image on the screen if the secure boot is disabled, according to the changelog. NetworkManager 1.38.4 and mutter 42.4 were also updated in the snapshot.

GNU’s collection of binary tools binutils 2.39 was the lone package to update in snapshot 20220813. The ELF linker now supports a --package-metadata option that allows embedding a JSON payload in accordance to the Package Metadata specification. The linker can also now generate a warning message if the stack is made executable.

Snapshot 20220812 had relatively few packages updated. The one major version update was made to the parse and domain-name decomposer rubygem-public_suffix 5.0.0. The new major version updated definitions and requires a minimum Ruby 2.6 version. The use of importlib-metadata for runtime package version lookups was made in the python-pbr 5.9.0 update, which is used to manage setuptools packaging. Another package to update in the snapshot was ncurses, which trimed out some unwanted linker options seen in Fedora 36.

The 20220811 snapshot started off the week with updates to Mozilla Firefox 103.0.2. The browser update fixed menu shortcuts for users of the JAWS screen reader and fixed an occasional non-overridable certificate error. The 42.4 version of gnome-desktop made Italian and Serbian translation changes and fixed detail text when it contained markup. An update of icewm 2.9.8 made a change that a restart will start icewm if no Window Manager is active and the package also updated the grouping menu when removing a task. Vim had its first update of the week in this snapshot and iproute2 5.19 added a set command and a group link with it ipstats. Intel had a CVE fixed in the ucode-intel 20220809 update; the company thanked those involved for helping find and solve CVE-2022-21233, which affected some processors.

Update, February 3rd 2025: It's been a couple of years, and by now, this article is extremely outdated. In fact, it was rendered obsolete roughly two weeks after its publication, when Stable Diffusion was released to the public. The technical bits aren't that interesting anymore, but I think the legal and ethics parts, such as they are, have held up.

I recently got access to OpenAI's DALL-E 2 instance. It's a lot of fun, but beyond its obvious application as a cornucopia of funny cat avatars, I think it's now fit to use in certain kinds of creative work.

There are already plenty of good articles out there on the model's strengths and weaknesses, so I won't go over that here other than to note that it's not a threat to high-end art. It's got an idea of what things look like and how they can visually fit together, but it's very vague on how they work (e.g. anatomy, architecture, the finer points of Victorian-era dinner etiquette, art critics), and object inpainting aside, it doesn't rise to the level of realism where I'd worry too much about the fake news potential either.

However, with human guidance and a carefully chosen domain, it can still do some very impressive things. I've suspected that adventure game graphics in the point-and-click vein could be one of those domains, and since I'm helping someone dear to me realize such a game, I had the excuse I needed to explore it a little and write this case study.

Inspiration

Point-and-click adventures make up a fairly broad genre with many different art styles. I've focused my attention on a sub-genre that hews close to the style of early 1990s Sierra and LucasArts adventure games. These would typically run at a screen resolution of 320×200 and appear pixelized, especially so on a modern display:

Contemporary game developers sometimes work at low resolutions, producing a similar effect:

At first glance this seems restrictive (just ask H.R. Giger), but from a certain point of view, it's actually quite forgiving and confers lots of artistic license:

• The perspective doesn't need to be realistic or even consistent, and is often tweaked for practical reasons, such as eliminating visual clutter, providing more space for the action or aligning better with the pixel grid.
• Speaking of pixels, pixelization helps work around the fact that DALL-E can produce odd smudges and sometimes struggles with details. It also helps with manual retouching, since there aren't very fine details or textures to be precisely propagated.
• Does your art look weird? Uncanny valley anxiety? Take a free tour courtesy of the entire genre. Feel your troubles float away as it throws an arm around you. And another arm. And another.

Ahem. What I'm trying to say is, this is a wonderful, fun genre with many degrees of freedom. We'll need them!

How to into the pixels

While you can tell DALL-E to generate pixel art directly, it's not even remotely up to the task; it just doesn't know how a pixel grid works. The result will tend to have some typical pixel art properties (flattened perspective, right angles, restricted palette with colors that "pop") wrapped in a mess of smudged rectangles of all sizes:

It's impressive in a "holy guacamole, it kind of understood what I meant" way, but even if you clean up the grid you don't stand a chance of getting a consistent style, and you have no control over the grid size.

Fortunately, pixelization can be easily split off from the creative task and turned over to a specialist tool. I used magick in my scripts:

$ magick -adaptive-resize 25% -scale 400% in.png out.png

It's worth trying different resampling filters. ImageMagick's -adaptive-resize operator produces nice and crisp output, but when downsampling by this much there may be even better options.

You could also experiment with color reduction and dithering. The images I generated for this article have been postprocessed like this…

$ magick -adaptive-resize 25% -ordered-dither checks,32,32,32 \
    -scale 800% in.png out.png

…which pixelizes to a 1:4 ratio, restricts the output to a color cube with 32 levels per channel (i.e. 15-bit color) and applies subtle — but not too subtle — checker-pattern dithering. It also upscales to twice the original size for easy viewing in a web browser.

Style prompts and selection

After some trial and error, I settled on a range of prompts involving techniques, styles and authors of fine art: oil on canvas, high renaissance, modernism, precisionism. This gave me a good chance of output in a handful of repeatable styles with sufficient but not overwhelming detail:

Beyond important details ("sunny day"), vague modifiers like "atmospheric", "dramatic" and "high quality" can have huge effects on lighting, camera angles and embellishment. They're also very unreliable, and I have the feeling they can crowd out more important parts of the prompt from the model's tiny mind and cause them to be overlooked. It's better to use compact, specific prompts until you're close, and then, well, watch it fall apart as you add a single modifier.

Which brings us to the second human-intensive part of this task: selection. Since the OpenAI UI produces four variants for each prompt, this is mandatory. It's also very necessary, as most of the output falls far short of the mark. With the right prompt, you might get a distribution where roughly 1/20 images is good (with minor defects) and 5/20 are potentially salvageable. The remainder will be obviously unusable for various reasons (major defects, stylistic and framing issues, photobombed by anthropomorphic utility pole).

I think it's the same way with the impressive DALL-E mashups being shared. By the time you're seeing them, they've been curated at least twice; once at the source, and one or more times by the chain of media that brought them to you. You won't see the hundreds of images that came out funny but not ha-ha funny.

Since each image takes only a second to generate and a few seconds to evaluate, this wild inconsistency isn't disqualifying. It just means DALL-E isn't magical or even very intelligent.

Setting the stage

An adventure game location is a bit like a theatre stage; it's a good idea to have an ample area close to the camera for the player to walk around in. It's also a good idea to avoid scenes where the player can get far away from the camera, as you'd be forced to choose between a comical perspective mismatch and a really tiny player character that'd be hard to follow and control. Obviously a real game won't want to follow these rules strictly, but it's important to be able to implement them when needed.

Fortunately it can be done, and it's not too hard:

To control the perspective and make it more flat, adding "facade" seemed to be effective. Ditto "diorama" and "miniature", although they tended to produce a more clinical look. Specifying a typical ground-level detail to focus on, e.g. "entrance", was also helpful. I'm not sure "2d" and "2.5d" actually made any difference. Bringing it all together:

• Specify the era, time of day and lighting conditions (e.g. "on a sunny day in the 2000s").
• Be specific about the overall location ("town", "city", or a named geographical location), the focus ("facade", "hotel entrance") and the immediate surroundings ("houses", "streets", "plains").
• You can explicitly ask for open space, e.g. "…and street in front" or "plaza surrounded by…".
• Sometimes it's necessary to ask for the space to be empty, otherwise DALL-E can paint in objects and people that you'd rather add as overlays later on.
• You can also specify camera placement, e.g. "seen from second-floor balcony", but you risk ground-level details becoming too small.
• Some combinations will have the model drawing blanks, resulting in ignoring much of your prompt or horking up non sequitur macro shots of blades of grass and the like. Be prepared to rephrase or compromise. Think about what might be well represented in the training set.
• Do not under any circumstance mention "video game", unless you want blue neon lights on everything.

Retouching and editing

This is easy to do using the in-browser UI. Just erase part of the image, optionally edit the prompt and off you go. Very useful for those times you've got something great, except there's a pine growing out of a church tower or an impromptu invasion of sea swine. Adding objects works too. Here's a villain's preferred mode of transportation, quite believable (if out of place) on the first try:

You can also upload PNG images with an alpha channel, although I had to click somewhere with the eraser before it would accept that there were indeed transparent areas. I suspect you could use this to seed your images with spots of color in order to get a more consistent palette.

Extending the images

DALL-E generates 1024×1024-pixel postage stamps. To fill a modern display you want something closer to a 19:10 ratio. Transparency edits come in handy here. The idea is to split the original image into left-right halves and use those to seed two new images with transparency to fill in:

This is easily scriptable. Note that you have to erase the DALL-E signature from the right half to prevent it bleeding into the result. Something like this can work:

$ magick in.png -background none -extent 512x0 -splice 512x0 left.png
$ magick in.png \( +clone -fill white -colorize 100 -size 80x16 xc:black \
    -gravity southeast -composite \) -alpha off -compose copy_opacity \
    -composite -compose copy -background none -gravity east -extent 512x0 \
    -splice 512x0 right.png

Upload left.png and right.png, reenter the prompt and generate a couple of variants for each. Since there's lots of context, the results turn out pretty good for the most part. Then stitch the halves together like this:

$ magick +append left.png right.png out.png

With a little more scripting, you can generate all possible permutations and apply your big brain judgement to them, e.g:

…and so on. You can also tweak the side prompts. Put in a pool or whatever:

I wouldn't be surprised if this kind of image extension made it into the standard toolbox at some point.

Other things it can do, and some that it can't

I had some success with interiors too. "Cutaway" was a handy cue to knock out walls and avoid claustrophobic camera placement, and it handled decorations and furniture fairly well (e.g. "opulent living room with a table and two chairs"). It could also generate icons for inventory items after a fashion ("mail envelope on black background"). I didn't delve very deeply into that, though.

You've probably noticed that the generated images all contain defects. Some can be fixed by erasing them and having DALL-E fill in the blanks, but others are too numerous, stubborn or minute for that to be practical. This means you'll have to go over each image manually before pixelization (for rough edits) and after (for the final touch). You'll also need to adjust colors and levels for consistency.

DALL-E can't write. In fact it will rarely be able to arrange more than three letters in a correct sequence, so if you want words and signage, you'll have to draw it yourself. Maps and other items that convey specific information by virtue of their geometry can probably also be ruled out, although you may get lucky using a mostly transparent cue sketch.

You won't get much help with animations, especially complex multi-frame ones like walk cycles.

If you want to convert an existing daylight scene into a night scene, that's probably best done manually or with the help of a style transfer model.

I realize I've barely scratched the surface here, and there's bound to be a lot more that I haven't thought of.

The economics of AI jackpot

OpenAI controls usage through a credit system. Currently, one credit can be used to generate four images from a single prompt, or three edits/variants from a single image and prompt. I got some free welcome credits (50 or so), and they're promising another 15 each month. When you spend a credit, it takes 4-5 seconds to get results, which means about a second per image. You can buy 115 credits for $15 + tax, which in my case works out to a total of $18.75. That's $0.163 per credit, or at most $0.0543 per image (batch of three).

Let's say you use this to generate locations for a point-and-click game. How many will you need? Well, one very successful such game, The Blackwell Epiphany (made entirely by the fine humans at Wadjet Eye Games), has about 70 locations. If you're considering AI-generated images for your game, you're probably not trying to compete with one of the industry's most accomplished developers, so let's lower that to 50.

50 locations is still a lot, and as I mentioned before, only 1/20 images come out adequate. For each location, you can probably get by with 10 adequate candidates to choose from. That means you'll generate 200 images per location, or 10,000 images total. Let's double that to account for some additional curation, edits, horizontal extensions, late changes to the script and plain old mistakes. Then, 20,000 * $0.0543 = $1,087. Since most of the images will be generated in batches of four, not three, it's fair to round that down to an even $1,000. It's probably not your biggest expense, anyway.

How about time investment? I mean, evaluating that many images seems kind of crazy, but let's do the math and see. If an image takes about 1s to generate and you spend about 5s deciding whether to keep it (recalling that 95% is quickly recognizable as dross and you'll be looking at batches of four), that's 20,000 * 6s = 120,000s or about 33 hours. Even if you can only stand to do it for two hours a day, you should be done in three to four weeks.

Throughout this you should be able to generate 10 candidates and 10 edits for each location. Further manual editing will likely take much longer than three weeks, but that's not something I'm experienced with, so I've really no idea. It also presupposes that you're starting out with a detailed list of locations.

Legal considerations

In addition to their API policies, OpenAI have public content policy and terms of use documents that appear to be specific to DALL-E. I'm not trained in law, but the gist of the content policy appears to be "don't be mean, sneaky or disgusting", which is easy for us to abide by with only landscapes and architecture. Some of the restrictions seem unfortunate from the perspective of entertainment fiction: Could I generate a bloodied handkerchief, a car wreck or something even worse? Probably not. Anything containing a gun? Certainly not. However, they're also understandable given the stakes (see below).

The most concerning thing, and likely a showstopper for some creative enterprises, is point 6 of the terms of use: Ownership of Generations. My interpretation of this is that generated images are the property of OpenAI, but that they promise not to assert the copyright if you observe their other policies (which may, presumably, change). If you're making a long-lived creative work, especially something like a game that may include adult topics alongside the generations, this seems like a risky proposition. I wouldn't embark on it without seeking clarification or some kind of written release.

Ow, my ethics!

So, yeah, ethics. An obvious line of questioning concerns misuse, but OpenAI is erring on the side of caution (or realistically, trying to keep the lid on just a little longer), and anyway, our use case isn't nefarious.

What's more relevant to us is the closed training dataset and how it might contain tons of formerly "open" but copyrighted material, or simply pictures whose author didn't want them used this way. We're talking half a billion images, and the relevant research and blog posts either allude to web scraping or mention it outright. A search for reassurance didn't turn up much, but I did discover an interesting open issue. So, could this be disrespectful or even abusive?

A common defense claims that the model learns from the training set the same way a human student would, implying human rules (presumably with human exceptions) should apply to its output. This can seem like a reasonable argument in passing, but besides being plain wrong, it's too facile since DALL-E is not human-like. It can't own the output (or, as the case would be, sign its ownership over to OpenAI) any more than a relational database could.

A better argument is that the training process munges the input so thoroughly that there's no way to reconstruct an original image. You don't have to understand the process deeply to see that this makes sense: there's terabytes of training data and only gigabytes of model. Then the implication becomes that this is transformative remixing and potentially fair/ethical use.

Thinking about this kind of hurts my head, particularly as it's also playing out in my own field. I haven't definitely concluded, but in general I think it's important to focus on the net good that technology and sharing can bring and how the benefits (and obligations) can be distributed equitably.

So is this going to upend everything?

Well, not everything. But some things, for sure. Neural networks have evolved very quickly over the past couple of years, and it looks like there's plenty of low-hanging fruit left. Current research leaves the impression that DALL-E 2 is already old news. There are also open efforts that seem to be completely caught up, at least for those with some elbow grease and compute time to spare.

A dear friend of mine joked that we've been privileged to live in a truly global era with minimal blank spots on the map and a constant flow of reasonably accurate information, the implication being that the not too distant past had mostly blank spots and the not too distant future will be saturated with extremely plausible-looking gibberish. We had a laugh about that, but you have to wonder.

A clarification regarding pixelization

August 18th: This article sort of made the rounds, and there has been lots of interesting feedback. Contrary to conventional wisdom, my experience when this happens is that other netizens are mostly very thoughtful and nice. So too this time.

There's one recurring criticism I should address, though, because I think it stems from my own carelessness and/or attempts at brevity: This is not pixel art!

Well, sure. It's not what we understand to be pixel art in 2022. Pixel art is when you hand-pixel the things, and maybe this could be too if it had gotten seriously retouched during/after pixelization — but in its current state it isn't. Since I'm aware of the distinction, I tried to word the article carefully ("adventure game graphics", "pixelization", "pixel graphics" in lieu of "lowres" or whatever) and only brought up pixel art in relation to the DALL-E query because, y'know, one can dream.

I sort of danced around it, though, and never explicitly said so. Here's the missing section:

An important reason I started out with a comparison to early 90s games (and not, say, any of the great 80s games) is that around that time, it became practical to make adventure game art using physical media (e.g. matte painting) and then digitizing it, which is somewhat similar to what I'm attempting here — except there's no need to digitize DALL-E's paintings (already digital). Instead, I added postprocessing as a callback to the traditional digitization process. Here's an excerpt from a nice Sierra retrospective by Shawn Mills. Go read the whole thing, it's great:

"We started painting with traditional media and then had our programming team develop some amazing codecs to scan the artwork in. That also allowed us to key scenes for other people and send them overseas to places like Korea and have them paint them. So we could really up the production because we [alone] never would have been able to raise the quality of production that we wanted."

Bill Davis, first Creative Director at Sierra On-Line

So Sierra got more for less, and they did this by sending prompts to dedicated overseas painters (vs. learning to paint faster/better or hiring more in-house). Sound familiar?

There's still the sort of "ugh" reaction one might have to talking about efficiency and art in the same sentence, and I do feel that. It's more fun to learn a new skill, or to work closely with others.

What's not fun, though, is to watch a nice, practical article that's already way too long start to overflow with the author's opinions and speculation. That's what the comments are for.

In the classic analog synthesizer, a sound is created by a simple oscillator and then carving/shaping the sound with filters. That’s why it’s sometimes called subtractive synthesis.

The sound effects in the Settings’ panel have been recreated with a method called Frequency Modulation where some oscillators aren’t used to generate the sound itself (carrier), but modulate it futher. Even here the significant role of filtering remains. In addition more complex sounds are achieved with so called layereing which is exactly what the name hints to, just using a bunch of sounds together in a mix.

Sounds created for GNOME 43 were generated on a mini-computer called Teensy (currently unavailable due to the global chip shortage), running software called Dirtywave Headless written by Timothy Lamb. The software includes other synthesizer engines, but majority of the sounds were made using the 4 operator FM engine. To further complicate things, my favorite algorithm is No.16 where all of the 4 oscillators are carriers, effectively being equivalent to a 4 oscillator analog synth.

Finally everything was cleaned up in Audacity.

To form a complete circle, and to my genuine surprise, my old friend Noggin from the Jeskola Buzz days has composed a great track using only samples from the gitlab issue (my involvement with music trackers predates GNOME or Free software in general. An old friend indeed).

Take a listen.

I wish I had published the project bundle to allow for easier adjustments, but better late than never.

Shorter Introduction

PR: https://github.com/VowpalWabbit/vowpal_wabbit/pull/4073

Tutorial: https://vowpalwabbit.org/docs/vowpal_wabbit/python/latest/tutorials/cmd_csv_with_iris_dataset.html

Introduction

My project here at the Reinforcement Learning Open Source Fest 2022 is to add the native CSV parsing feature for the Vowpal Wabbit.

So why do I choose to implement native CSV parsing? CSV is one of the most popular file formats used in the machine learning dataset and is often delivered as the default format in competitions such as Kaggle. CSV files have the same schema for each example, while VW text format doesn’t. Although converters in Python and Perl have been written which convert these files to VW text format, it would be convenient if VW could understand CSV files natively. I also want to challenge myself, as there is a surprising amount of complexity in the design of implementing a generalized parser for CSV, so this project is as much about considering all of the design pieces as implementing a working parser. My choice and time devoted also pay off since my project has already got merged into the VW upstream !!!

About CSV

CSV files are often separated by commas (,) or tabs. however, alternative delimiter-separated files are often given a .csv extension despite the use of a non-comma field separator. This loose terminology can cause problems in data exchange. Many applications that accept CSV files have options to select the delimiter character and the quotation character. Semicolons (;) are often used instead of commas in many European locales in order to use the comma (,) as the decimal separator and, possibly, the period (.) as a decimal grouping character.

Separating fields with the field separator is CSV’s foundation, but commas in the data have to be handled specially.

How do we handle CSV files?

The short answer is that we follow the RFC 4180 and MIME standards.

The 2005 technical standard RFC 4180 formalizes the CSV file format and defines the MIME type “text/csv” for the handling of text-based fields. However, the interpretation of the text of each field is still application-specific. Files that follow the RFC 4180 standard can simplify CSV exchange and should be widely portable. Here are its requirements:

1. Lines will end with CR or CRLF characters, and it is optional for the last line.
2. CSV files can have an optional header record. There is no sure way to detect whether it is present, so care is required when importing.
3. Each record should contain the same number of separated fields.
4. Any field may be quoted with double quotes.
5. Fields containing a double-quote or commas should be quoted.
6. If double-quotes are used to enclose fields, then a double-quote in a field must be represented by two double-quote characters.

Dig into details

1. Allows specifying the CSV field separator by –csv_separator, default is ,, but " | or : are reserved and not allowed to use, since the double quote (") is for escape, vertical bar(|) for separating the namespace and feature names, : can be used in labels.
2. For each separated field, auto remove the outer double-quotes of a cell when it pairs. --csv_separator symbols that appeared inside the double-quoted cells are not considered as a separator but a normal string character.
3. Double-quotes that appear at the start and end of the cell will be considered to enclose fields. Other quotes that appear elsewhere and out of the enclose fields will have no special meaning. (This is also how Microsoft Excel parses.)
4. If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote, and will remove that escape symbol during parsing.
5. Use header line for feature names (and possibly namespaces) / specify label and tag using _label and _tag by default. For each separated field in header except for tag and label, it may contain namespace and feature name separated by namespace separator, vertical bar(|).
6. --csv_header to override the CSV header by providing the header. Combined with --csv_no_file_header, we assume that there is no header in the CSV file and under such condition specifying --csv_header for the header is a must.
7. If the number of the separated fields for current parsing line is greater than the header, an error will be thrown.
8. Trim the field for ASCII “white space”(\r\n\f\v) as well as some UTF-8 BOM characters(\xef\xbb\xbf) before separation.
9. If no namespace is separated, will use empty namespace.
10. Separator supports using \t to represent tabs. Otherwise, if assigning more than one character, an error will be thrown.
11. Directly read the label as string, interpret it using the VW text label parser.
12. Will try to judge if the feature values are float or string, if NaN, will consider it as a string. quoted numbers are always considered as strings.
13. If the feature value is empty, will skip that feature.
14. Reset the parser when EOF of a file is met (for possible multiple input file support).
15. Support using --csv_ns_value to scale the namespace values by specifying the float ratio. e.g. –csv_ns_value=a:0.5,b:0.3,:8, which the namespace a has a ratio of 0.5, b of 0.3, empty namespace of 8, other namespaces of 1.
16. If all the cells in a line are empty, then consider it an empty line. CSV is not a good fit for the multiline format, as evidenced by many empty fields. The multiline format often means different lines have different schemas. However, I still leave the empty line support to ensure that it’s flexible and extendable. In this case, users can still express multiline examples in CSV files, although it is not listed as supported. We still throw an error if the number of fields separated by the line doesn’t match the previous, even if all the fields are empty, as this usually means typos that users may not intend.

Some statistics to share

The project reaches 100% Test and Code Coverage. On my computer, it takes only 829 ms to handle 200000 examples, which is comparable to the equivalent VW custom format data file’s parsing speed of 623 ms, and is equivalent to 21.7 MB/s throughput.

I have expected that little bit of slower between VW custom text format and the CSV format. The VW text parser can parse as it reads since generally the elements have a fixed position, while CSV parser needs to store the split elements into an array, and look up that array according to the header for labels, tags, namespace and feature values, also there’s double quotes trimming and escape support, which will definitely cost more time.

After all, I have tried my best to optimize it and the performance is also to my satisfaction. You can check using the throughput tool for the unoptimized one during the project. And that was actually 10 times improvement!

• VW text Parser:

2239418 bytes parsed in 60581μs
36.9657MB/s

• Unoptimized CSV Parser:

1799450 bytes parsed in 912927μs
1.97108MB/s

• Optimized CSV Parser:

1799450 bytes parsed in 87728μs
20.5117MB/s

Credits

That’s all, thanks for reading, and also special thanks to my mentors Jack Gerrits and Peter Chang from Microsoft Research – New York City Lab, who really helped me a lot during the project.

Reference

1. Comma-separated values – Wikipedia

For 2022 final result, I passed the final evaluation, obtained all the honor titles (Only 2/4000+ in all participants): Tencent Open Source Contributor Certificate (only issued 30+ around the globe at that time), Intern of Excellence, Task Scholarship.

Issue

Task 1

Requirement

https://github.com/openjdk/jdk/pull/9541

Task 2

Requirement

Result

Tested computing the signature as well as verifying the signature for comparing between secp256r1 and secp256k1 using SHA256withECDSA with the help of the SunEC provider.

The result clearly shows that secp256r1 has a better performance than secp256k1 with regard to SHA256withECDSA when signing. But secp256r1 has almost the same performance as secp256k1 when verifying.

Benchmark                                 Mode  Cnt     Score    Error  Units
BenchmarkSigning.secp256k1_1024B         thrpt   25  1237.482 ± 30.518  ops/s
BenchmarkSigning.secp256k1_1024K         thrpt   25   318.589 ±  2.656  ops/s
BenchmarkSigning.secp256k1_128B          thrpt   25  1266.561 ± 32.311  ops/s
BenchmarkSigning.secp256k1_256B          thrpt   25  1254.327 ± 36.935  ops/s
BenchmarkSigning.secp256r1_1024B         thrpt   25  1746.453 ± 33.358  ops/s
BenchmarkSigning.secp256r1_1024K         thrpt   25   340.530 ±  3.970  ops/s
BenchmarkSigning.secp256r1_128B          thrpt   25  1763.460 ± 31.179  ops/s
BenchmarkSigning.secp256r1_256B          thrpt   25  1756.899 ± 32.715  ops/s
BenchmarkVerifying.secp256k1_1024B       thrpt   25   486.545 ± 13.410  ops/s
BenchmarkVerifying.secp256k1_1024K       thrpt   25   228.638 ±  2.478  ops/s
BenchmarkVerifying.secp256k1_128B        thrpt   25   491.065 ± 13.948  ops/s
BenchmarkVerifying.secp256k1_256B        thrpt   25   499.466 ±  4.558  ops/s
BenchmarkVerifying.secp256r1_1024B       thrpt   25   402.902 ± 53.489  ops/s
BenchmarkVerifying.secp256r1_1024K       thrpt   25   212.743 ± 23.301  ops/s
BenchmarkVerifying.secp256r1_128B        thrpt   25   401.215 ± 65.401  ops/s
BenchmarkVerifying.secp256r1_256B        thrpt   25   393.021 ± 79.755  ops/s

Further investigation shows that before secp256k1 was removed from JDK, all the curves seem to be realized by C using the OS library instead of Java. https://github.com/openjdk/jdk/blob/jdk-11+28/src/jdk.crypto.ec/share/native/libsunec/impl/oid.c#L95

JDK-8238911 in 2020 reported the weaknesses in the implementation of the native library EC code make it necessary to remove support for future releases. The most common EC curves (secp256r1, secp384r1, and secp521r1) had been re-implemented in Java in the SunEC JCE provider.

After some communications with my mentor Johns Jiang, he tells me that JDK-8181594 introduces the optimized finite field implementations in Java. Previously before that implementation was introduced, pure Java realization was really slow, then we use the OS library to realize all the curves so that the performance can be improved. But now, instead, with the help of that optimized Java library, Java realization takes the advantage and becomes the most efficient one, it’s now even comparable with the pure C realization.

Our flame graph also some kind confirms this, as you can see here, the Java Flight Recorder (JFR) can record the secp256r1 methods calling stacks, but it’s not the case for secp256k1. So it’s likely that secp256r1 has a better performance than secp256k1 for signing since it’s fully realized in Java and using that optimized library, thus reduces the calling costs for the OS library. If they are both realized in Java using the optimized method, I guess there should be no difference.

As secp256k1 has already been removed in JDK and now secp256r1 does have a better performance, so I guess here we will have no obvious further room for improvement.

Task 3

Requirement

The final report (in Chinese) is not limited to the following content, it also includes some more security analysis for different ways of realization.

With modification of SunEC provider

Result

As for Elliptic-curve based cryptography algorithms, the curve parameters are used to generate the keys. The official recommended curve parameters for SM2 can be seen here: https://www.oscca.gov.cn/sca/xxgk/2010-12/17/1002386/files/b965ce832cc34bc191cb1cde446b860d.pdf

That curve parameters is also known as sm2p256v1, since it’s also a prime curve just like secp256r1, we can use the existing implementation of secp256r1 in the SunEC library to help us realize our implementation.

The OID for sm2p256v1 is 1.2.156.10197.1.301: http://gmssl.org/docs/oid.html

https://github.com/HollowMan6/jdk/tree/sm2 https://github.com/HollowMan6/jdk/commit/c3e924641bb3a838f6abc496dd380ceb619df163

We first fill the curve parameters into the CurveDB

Then add the OID and names.

The most important part is FieldGen. FieldGen is used to automatically generate optimized finite field implementations, which is also the library I mentioned in Task 2 JDK-8181594 for improving Java version’s efficiency. https://github.com/HollowMan6/jdk/blob/c3e924641bb3a838f6abc496dd380ceb619df163/make/jdk/src/classes/build/tools/intpoly/FieldGen.java We need to generate two fields, Integer Polynomial (corresponds to parameter p) and Order Field (corresponds to parameter n).

As:

FFFFFFFE FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 00000000 FFFFFFFF FFFFFFFF

=

FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF $2^{256} – 1$

–

00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000 $2^{224}$

–

00000000 00000000 00000000 00000000 00000000 FFFFFFFF FFFFFFFF FFFFFFFF $2^{96} – 1$

+

00000000 00000000 00000000 00000000 00000000 00000000 FFFFFFFF FFFFFFFF

$2^{64} – 1$

= $2^{256} – 2^{224} – 2^{96} + 2^{64} – 2^0$

So the Integer Polynomial shall fill just like that. We can copy other parameters from secp256r1 as they are all 256 digit prime curve.

The private keys in Hex can be printed directly with no special format.

Since the public keys in Hex can be compressed, it does have a special format, that if it starts with 04, then the keys are uncompressed and we just then concat the X and Y coordinate together. The compressed ones always start with 02 or 03, and then only the X coordinate is needed. The 02 and 03 is determined by that, when Y coordinate is even, we use 02, use 03 when odd. In addition, we have to also make sure that both X and Y coordinates are 64 in length for Hex.

As the Bouncy Castle library has already fully realized the SM2, to ensure that the generated keys fit the sm2p256v1, I also use the generated keys for signing using SM3withSM2. The validity of the keys can be verified during the signature verification processes, during which we recover the sm2p256v1 elliptic curve point from the Bouncy Castle library based on the Hex format public key. If we use keys generated based on other curves, like secp256r1, error will be thrown.

Demo result:

Public Key (Uncompressed): 040A8FB35CA4761FAAA36B2A24E77EC657D96F74147C50EE2D5B50E3AAFD8304D8CBB65FB2E661D37B7C3B900E1BDBEDE894D9CBB9079E8DD704B9465BFF65EE17
Public Key: 030A8FB35CA4761FAAA36B2A24E77EC657D96F74147C50EE2D5B50E3AAFD8304D8
Private Key: 42756D22960A58D08F9E7E3A0D56D9630D8D051D082F4D2BFCE22FD2653524EB
To sign: How are you?
Signed: MEUCIAgvYgl0ydHwd536MkmwaRuhmkD/klh79VmHEBJI1zCRAiEAo9jNkGM+Tjh/0AmX82nSPOMYgRPaWm6SUXiB63YGAD4=
Verification OK!

Error thrown when using secp256r1 (faulty key pairs):

Public Key (Uncompressed): 0456A3205C7B47BF303F4B65CDAB5B94C343BE7220E5AAAB001B7263DBCD113B42447A9E41BF1374D4ABC7A2AE31E7441E3EB20D5808CCB7D88BFE4F8F2C9887C3
Public Key: 0356A3205C7B47BF303F4B65CDAB5B94C343BE7220E5AAAB001B7263DBCD113B42
Private Key: A33048CC39925B5D4BA4C34FE846C85D41DA5AABA0CFDE4092A7A4ED716D557
To sign: How are you?
Signed: MEYCIQCktncEmfLbC9rLC1Im9gj/AvZRUIQ5Z1plrq1X0L/5YwIhAOa0JeSoQFnV51kAJsFRY3T4cpCn2O7bKoN+M+nPpv6y
Exception in thread "main" java.lang.IllegalArgumentException: Invalid point coordinates
        at org.bouncycastle.math.ec.ECCurve.validatePoint(Unknown Source)
        at org.bouncycastle.math.ec.ECCurve.decodePoint(Unknown Source)
        at org.sample.SM2Util.verify(SM2Util.java:103)
        at org.sample.SM2Util.main(SM2Util.java:129)

JMH Performance test for compressed and uncompressed public key generation result shows that the compressed public keys generation has a better performance than the uncompressed ones. You can find the reason from the flame graph of Java Flight Recorder (JFR), that it’s because the uncompressed ones also need to caculate the Y coordinate Hex format, which takes a lot of time.

Benchmark                                         Mode  Cnt       Score       Error  Units
BenchmarkPublicKeys.sm2p256v1_compressed         thrpt   25  1212201.531 ± 248181.084  ops/s
BenchmarkPublicKeys.sm2p256v1_uncompressed       thrpt   25   760033.805 ±  35058.515  ops/s

Homemade

Our realization refers to the JavaScript implementation here, Wikipedia Elliptic curve point multiplication and the official documentation .

The homemade one is based on purely mathematical methods, no other dependencies.

We use SecureRandom to generate the private key, we also tried public key generation using the Standard Projective Coordinates and Jacobian Coordinates, as well as the binary expansion and addminus algorithm.

See result for more performance comparisons.

JMH Performance test for compressed and uncompressed public key generation result shows that the uncompressed public keys generation has almost the same performance as the compressed ones. Though it seems like a contradiction, here the uncompressed Y coordinate Hex caculation counts significantly smaller when you take the overall time into consideration.

More about 2023 Blockly Summit

Project Repository Link

Try out the plugin online

PRs and issues resolved during GSoC 2022:

• PR #1
• PR #2
• Issue #3
• PR #4
• PR #5
• google/blockly-samples PR #1202
• ThibaultJanBeyer/DragSelect PR #128

Public Announcement

My GitHub

Introduction

Hi there! If you prefer watching videos, check this out for an episode demoing the plugin:

For you guys who prefer reading, in this blog post I will introduce you to my work on the plugin for selecting, dragging, and doing actions on multiple blocks. The project is sponsored by Google Summer of Code (GSoC) 2022, under MIT App inventor.

Backgrounds

Aim

The project aims to enable the selection of multiple blocks at the same time and allow moving and doing actions on multiple blocks on Blockly.

This behavior is the same as when you try to manage your files on your Operating System. You can click on the files while pressing the control key, drag a rectangle to select multiple files. Then move them around, copying and deleting.

It sounds a little bit easy, but actually, I would say that it’s not. Multiple selections can become a crazy-complex feature when you start thinking about the details.

History

This feature request has remained open on GitHub Issues for six years. However, it was still in the discussion phase and far from the beginning of Implementation before my project began.

Since the Blockly community long wants this feature, we base our plugin on the latest Blockly so that it can be applied to everyone’s project. The App Inventor uses a Blockly version that is much older, so it’s a pity that we can’t see it work on App Inventor now. Let’s hope that the App Inventor can upgrade the Blockly version to the latest soon.

Realization

The “drag a rectangle to select” feature is realized with the help of the DragSelect plugin. I submitted a PR to add the pointer-events so that it can work for Blockly, and it got merged into v2.4.4.

In addition, I disable the drag surface feature in Blockly, which stops us from moving multiple blocks simultaneously. Also, there’s evidence suggesting that we can perform better without a drag surface.

So, how does the plugin work? Well, generally, the plugin acts like an adapter. It maintains its own multiple selection set, which keeps currently selected blocks and make sure we always have one of the selected blocks as the selected one in Blockly core. When users do some actions, the plugin also passes all the actions to the other blocks in our set besides the selected one in Blockly core.

Functionalities

Let’s check out what the plugin can do!

1. Additional blocks can be selected by holding the SHIFT key while clicking the new block. You can also deselect the block by clicking the already selected block.
2. Clicking on the button above the trashcan is equivalent to holding or releasing the SHIFT key for switching between the multiple selection mode and the normal mode.
3. We can clear the selection by clicking on the workspace background.
4. Clicking a new block without holding SHIFT key can clear the multiple selections and change the selection to only that block.
5. Holding SHIFT key to drag a rectangle area to select can reverse their selection state for the blocks touched by the rectangle.
6. In multiple selection mode, workspace dragging and block dragging will all be disabled. You can only drag to draw a rectangle for selection.
7. When some of the selected blocks are in one block stack, for example, some top blocks and some of their children blocks are in the selection simultaneously. If applicable, the plugin only disconnects the selected most top block in that stack with its parent block. Move along with all the children’s blocks of that most top block as a whole.
8. You can also drag all the blocks to the trash can.
9. When you edit the fields while selecting multiple blocks, we will automatically apply that to all the blocks with the same type.
10. There’s also an MIT App Inventor-only feature that has been migrated into this plugin, that you can double click to collapse or expand currently selected blocks.
11. For the context menu, the Duplicate will duplicate the selected most top block in the block stack and all the children blocks of that most top block. The selection will be changed to all newly created duplicate blocks’ most top blocks. For all the other items, The actions to show are determined by the state of the block which the user right-clicks on, and the same action will be applied to all the blocks no matter their individual state. We will append the currently applicable number of user-selected state-changing blocks, and the number will only be shown when it is greater than 1.
12. The Add Comment / Remove Comment option will add / remove comment buttons to all the selected blocks.
13. The Inline Inputs / External Inputs option will convert the input format with all the selected blocks.
14. The Collapse Block / Expand Block option will only apply to the selected most top block in the block stack.
15. The Disable Block / Enable Block option will only apply to the selected most top block in the block stack. All the children blocks of that most top block will also get disabled.
16. The number in Delete [X] Blocks is the count of the selected most top block in the block stack as well as all children of those selected most top block. Clicking on that option will delete the blocks mentioned.
17. The Help option displays just the helping information of the block the user just right-clicked on.
18. We add Select all Blocks in the workspace context menu.
19. For the shortcut keys, These actions will only apply to the selected most top block in the block stack. when you press Ctrl A, you can select all the blocks in the current workspace. Ctrl C to copy the selected blocks, Ctrl X to cut the selected blocks to the clipboard, and Ctrl V to paste all the blocks currently in the clipboard and get all the newly pasted blocks selected.
20. Bumping neighbors after dragging to avoid overlapping is disabled in this plugin by default, since I find it disturbing sometimes for multiple selections.
21. Click on a block will bring that block to the front in case Bumping neighbours is disabled.

Usage

If you want to integrate the plugin into your project, you can add the dependency into your package.json. In the source code, pass the workspace to the plugin, and initialize the plugin, and then it’s done! You can choose to disable double-click the blocks to collapse or expand and enable bumping neighbors after dragging to avoid overlapping. For the multi-select controls, you can also choose to hide the icon and customize the icons of each state. More details can be found in the README.

You can also choose to integrate the plugin with other ones, like the scroll options, which enable the edge scroll and wheel scroll for Blockly. The only thing you have to pay attention for the scroll options plugin to work is to assign the original blockDragger value required for scroll options to baseBlockDragger in our plugin. During the project period I also submitted a PR for fixing the a bug that makes scroll options unable to work without the drag surface, and it has already got merged.

Finally

That’s all for this blog post. Before it ends, I would like to say thank you to my mentors, Evan Patton and Li Li, as well as the MIT App Inventor Team for guiding me throughout the project. They are really supportive. Also, special Thanks to Beka Westberg. She devoted a lot of time to giving suggestions and helping review the code. We can’t have this plugin without her! Finally, thanks for reading this blog post!

Synopsis

Before the Summer of Bitcoin project, Cryptoanarchy Debian Repo (CADR) lacked Continuous Integration (CI), which troubles the new coming contributors because setting up the developing environment can be complex. I finally successfully implemented the CI using GitHub Actions default runners. The CI can be triggered manually, or by sending PRs as well as pushing directly to the master branch.

The CI is divided into 2 jobs:

• The first one is the build job. It builds the running podman environment image, uploads the image to Artifacts for reuse. Then with the podman environment, builds the CADR packages, checks their sha256 sum, and then uploads the built Debian packages as well as their sha256 sum values to Artifacts.
• The second one is the test job. It runs when the build job finishes. The testing jobs are run in parallel for each package. The test job first downloads the built images and packages Artifacts uploaded in the build job, then use make test-here-basic-% and make test-here-upgrade-%(% for package name) to run tests.

Road of Implementation

First try

At first, I ignored the fact that there already exists a dockerfile for CADR running (although it was not for building), and setup my own dockerfile from scratch by adding dependencies when I encounter any errors.

My own dockerfile turned out to work fine on GitHub Actions for the building process, but failed for the test process. Initially I thought it was some more dependency issues, since the test can work on my own computer.

When checking the logs I found that it’s due to the unshare issue.

Then I noticed that adding a --privileged parameter can fix the unshare issue for docker. But then a systemd issue just came after it. Finally I noticed the dockerfile in the codebase that already exists for CADR, and just as how it works, I made the systemd to start as the first process.

VOLUME [ "/sys/fs/cgroup" ]
CMD ["/lib/systemd/systemd"]

and add --tmpfs /tmp --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:ro as additional parameters to share some host machine resources and make it as the daemon container. Operations are done using docker exec to attach to the container. But still it doesn’t fix the systemd issue.

Finally I switched to podman and the systemd issue got fixed (it supports --systemd=true) because I occasionally found this article when I Googled the issue.

However, a new issue occurs suggesting failed to override dbcache.

Can’t see any errors from the workflow logs, even though the missing bc dependency has been fixed and I can confirm that the command here works correctly in the container running on my PC. Moreover when I manually skip the test just mentioned, the test further shows that there is an unneeded reindex:

Which suggests that it’s related to issue 108 maybe? Have opened an issue here. I can skip that test, but further error suggesting electrs is not available. It can be fixed with this PR

Even if those issues are fixed, another platform may still be needed since the full test requires too much space:

Only testing the regtest part will be fine with no space issue.

Fixing the tests

The first issue is to find out why overriding dbcache fails. This takes me quite a few weeks to find out the solution. Originally, when determining bitcoin dbcache size, bc is invoked. But bc is not guaranteed to be installed, thus it can fail. Then instead of doing maths wizardry, I submitted a PR to just match on ranges: RAM < 1024 -> default dbcache, RAM < 2048 -> dbcache=512, …, RAM => 8192 -> dbcache=4096. However, the failed to override dbcache still exists when running in GitHub Actions workflow with this PR. It’s very weird, as when I run the test manually using the same methods from the workflow with the podman container on my PC, such an issue won’t appear. So the issue seems to belong to GitHub Actions workflow environment only even though it’s running in the container.

Then after a lot of trial and error, I finally found the culprit. By referring to PR, previously the debcrafter dropped all capabilities which seems to cause errors when the host kernel capabilities do not match those known to setpriv. We need to only drop supported capabilities by the current kernel.

Then I submitted a PR to fix this, and got merged.

The second issue is to find out why the unneeded reindex error exists. I noticed that although the test failed on test-here-basic-electrs with unneeded reindex when running make test, running make test-here-basic-electrs alone won’t fail. So prior to fixing this issue, my guess was that maybe it’s still related to issue issue 108, which is that after updating existing non-pruned nodes in experimental -reindex was used despite pruning not being changed. Or the test environment didn’t get cleaned up before test-here-basic-electrs when executing make test. My final result proves that the later is right, and submitted a PR to clean up the chain mode marker, since we want it to be clean with package_clean_install.sh

I also submitted a PR to force linked-hash-map version to be 0.5.4 for fixing the build issue of debcrafter that just came up during the project period.

The test can be successful on GitHub Actions when only tested with regtest and after PR 200, 201, 203, 204 get merged.

Research on cloud provider

I also researched on which cloud service to use, as we were intended to use a GItHub self-hosted runner. As I checked on cloud service providers AWS, Google Cloud and Azure. I find that GitHub Actions uses Azure as their default runners, to build a GitHub self-hosted runner, Azure would be a good choice if we use the same provider as the GitHub default hosted runner. Also Azure has a trial period with 200 dollars for one month when starting a new account, so we can start free, while other cloud service providers don’t offer such a great discount.

Investigation on other possible CI platforms

I find out that unlike GitHub Actions that runs CI in a virtual machine, Travis CI, GitLab CI/CD, Jenkins all run the CI predefined Docker containers without without systemd support. Then it would be not possible to run a systemd supported podman container inside such a container. Azure DevsOps is a valid one for our use case, and I also tried to use it. Then I noticed that for a single CI job, each step, it only allows 1 hour maximum, otherwise it would get cancelled, while our build job as well as full net test job last much longer than 1 hour, it’s also not suitable for us. Then finally the GitHub Actions would be the only good choice to have.

Self-hosted runners for GitHub Actions

I had also tried to use Azure Virtual Machines service by myself to set up Self-hosted runners for GitHub Actions, but it seems like the environment doesn’t automatically get cleaned each time, and is contaminated from previous runs.

Divide and Conquer

Finally I came up with a new way for testing. We can use the divide and conquer to bypass the short of disk space issue as well as the contaminated running environment issue when running tests on both the mainnet and regtest. I can see that the make test is composed of make test-here-basic-% and make test-here-upgrade-%(% for package name), we can just use the matrix in GitHub Actions to test each package in separate environments, and then the disk space would be enough, and PR 204 can be closed because we now always have a clean environment.

Then I successfully implemented and tested that method, and it works! So right now it’s the full test and I have reached the project goal with even a solution that has no money cost for CI building and testing!

The CI can be successful if the previous tests fixing PRs get merged first.

Final Deliverables

After my mentor @Kixunil‘s review, I began to use --locked parameter to make cargo use cargo.lock which contains correct versions. Also I fixed a security issue, make a user account user and built/tested with user, and made to upload the sha256sum result for the built deb packages afterwards so people can check hashes.

In addition, I fixed tests for bitcoin-regtest after setting bitcoind nosettings enabled in PR 205. Then there should be some kind of bug in core for this wallet location difference in the GitHub runner and the physical machine, so the PR gets closed and another solution that makes tests independent of wallet location was committed.

The CI runs successfully with the above changes, and everything works fine, nothing left to do.

Conclusion

My Summer of Bitcoin project is a great experience for me as I learned a lot about Bitcoin and related DevOps knowledge.

If you are interested in Bitcoin and would like to start contributing to Cryptoanarchy Debian, with my work, you can easily fork the repo and add more test or new packages by committing the code to your fork’s master branch on GitHub, the CI will help you build the deb packages and locate possible errors, no need to setup the developing environment on your computer again.

Summary of PRs

Merged

• [SoB] Add GitHub workflow for CI
• Fix libcap-ng is too old for ‘all’ caps
• Remove bc dependency for overriding dbcache
• Change all apt into apt-get in test
• Try more times to test connecting to electrs-regtest
• Set bitcoind log file under /var/log
• Set bitcoind nosettings enabled by default

Closed as resolved in another way

• (Issue) Bug: Test failed for bitcoin-regtest with error unneeded reindex for test-here-basic-electrs
• Fix unneeded reindex error running tests
• Fix tests for bitcoin-regtest after setting bitcoind nosettings enabled
• Force linked-hash-map version to be 0.5.4
• rustyline downgrade to 6.3.0 to fix compiling issue