Ceph Meetup Berlin - Followup: librmb

Unicsy phone
But I realized Linux kernel is not really the most important part. There's more to Unix: compatibility with old apps, small programs where each one does one thing well, data in text formats so you can put them in git. Maemo got some parts right, at least you could run old apps in a useful way; but most important data on the phone (contacts, calendar) were still locked away in sqlite.
And that is something I'd like to change: phone that is ssh-friendly, text-editor-friendly and git-friendly. I call it "Unicsy phone". No, I don't want to do phone `cat addressbook | grep Friend | cut -f 1`... graphical utilities are okay. But console tools still should be there, and file formats should be reasonable.
So there is tui project, and recently postmarketos project appeared. Nokia N900 is mostly supported by mainline kernel (with exceptions of bluetooth and camera, everything works). There's work to be done, but it looks doable.
More is missing in the userspace. Phone parts need work, as expected. What is more surprising... there's emacs org mode, with great calendar capabilities, but I could not find matching application to display data nicely and provide alerts. Situation is even worse for contacts; emacs org can help there, too, but there does not seem to be agreement that this is the way to go. (And again, graphical applications would be nice).
Fibre Channel over Ethernet: Basics of FCoE in SUSE Linux
I had to apply a fix for a FCoE module in YaST, and I had no idea.
After learning a couple of things I still only have a vague idea, but I am writing it down to help my future self, my team mates, and perhaps you too.
FCoE stands for "Fibre channel over Ethernet". Apparently if you have some disk on a Fibre Channel SAN (storage area network), you can use FCoE to extend the reachability of that disk to the ethernet parts of your network. It still needs to be a kind of special ethernet (10Gb, with special network cards) but that seems less special than FC hardware.
For a better overview, including a diagram, see: SUSE Linux Enterprise Server Documentation / Storage Administration Guide / Fibre Channel Storage over Ethernet Networks: FCoE.
FCoE typically uses a virtual LAN, (VLAN, IEEE 802.1Q).
There needs to be a Fibre Channel Forwarder (FCF) between the FC and ethernet parts. It has a MAC address. Note a difference from iSCSI which works on the IP level, one layer up.
YaST helps you set things up. The rest of this article could be useful if you cannot use YaST for some reason.
SLES uses open-fcoe. On SLES-12 the package is called fcoe-utils.
fipvlan (stands for FCoE Initialization Protocol VLAN discovery) shows FCFs and which interface and VLAN they are reachable with:
# fipvlan --auto
Fibre Channel Forwarders Discovered
interface | VLAN | FCF MAC
------------------------------------------
eth1 | 500 | 00:0d:ec:b3:ca:00
It can also --create the VLAN interface and --start up the FCoE connection, but it won't make that permanent for the next boot
To make it permanent you need to
- enable the FCoE service (SLE11:/etc/init.d/boot.fcoe, SLE12: fcoe.service). Under the hood it uses two programs:
fcoemonis the daemon,fcoeadmis a front end (fcoeadm -pshows the pid offcoemon). - write a config file,
/etc/fcoe/cfg-*IFACE*, where IFACE is- eth1.500 if
AUTO_VLANisno; in this case, you also need/etc/sysconfig/network/ifcfg-eth1.500, seeman ifcfg-vlan. - eth1 if
AUTO_VLANisyes; in this case, the interface is namedeth1.500-fcoe. Note the unusual-fcoesuffix!
- eth1.500 if
With the config files in place, rcfcoe start (and ifup eth1.500, unless AUTO_VLAN). Then you should see the disk devices:
# fcoeadm --target
Interface: eth1.500
Roles: FCP Target
Node Name: 0x50060160BB600160
Port Name: 0x500601663B600160
Target ID: 0
MaxFrameSize: 2048
OS Device Name: rport-2:0-2
FC-ID (Port ID): 0x710D00
State: Online
LUN ID Device Name Capacity Block Size Description
------ ----------- ---------- ---------- ----------------------------
0 /dev/sdb 16.00 GiB 512 DGC VRAID (rev 0430)
[...]
People who actually know their way around FCoE will note that I have omitted many important details. Let me know in the comments whether I should come back to this and expand on some topics.
openSUSE with sudo – but convenient!
If you are used to handling activities with administrator rights (“root”) like I am from the Debian world, you will have some difficulties with openSUSE in the beginning. With two users it is still possible, because you can set the same password for both user and root. But at the latest with more user accounts this is already over, unless you give the root password to everyone. Both solutions are certainly somehow practicable, but it’s not very nice. Especially since sudo would actually be installed – but only halfway through.
So I started with my current openSUSE Tumbleweed to teach the system a reasonable sudo concept and then apply it to YaST. It was a bit nasty to find out, but in the end it worked well.
Let’s go!
visudo
By default sudo asks for the root password. This is pretty nonsensical, so let’s change it!
- In the first part we still work as normal users. The line details may vary depending on the age of the file/system version and previous changes to it.
sudo visudo
- The parameters in line 43 starting with env_keep = “LANG… at the end within the quotation marks:
DISPLAY XAUTHORITY
- Comment out lines 68 and 69 completely, so that the password of the “target user” is no longer requested:
#Defaults targetpw #ALL ALL = (ALL) ALL
- Additionally you uncomment line 81, so delete the comment character #:
%wheel ALL=(ALL) ALL
- Save, close and then add your user(s) to the group “wheel” either via YaST or directly in the terminal:
gpasswd -a <dein-username> wheel
By logging out and in again, the change will be applied and sudo wants to have your user password in the terminal from now on.
YaST
For the graphical version of YaST, PolicyKit is used for authentication, a little more work is needed here. From here on, you work as root, so change the account with su –.
- Create a PolicyKit Action for YaST
vim /usr/share/polkit-1/actions/org.opensuse.pkexec.yast2.policy
- Insert the following XML block into the file. Please pay attention to line breaks when copying/pasting.
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE policyconfig PUBLIC "-//freedesktop//DTD PolicyKit Policy Configuration 1.0//EN" "http://www.freedesktop.org/software/polkit/policyconfig-1.dtd"> <policyconfig> <action id="org.opensuse.pkexec.yast2"> <message>Authentication is required to run YaST2</message> <icon_name>yast2</icon_name> <defaults> <allow_any>auth_self</allow_any> <allow_inactive>auth_self</allow_inactive> <allow_active>auth_self</allow_active> </defaults> <annotate key="org.freedesktop.policykit.exec.path">/usr/sbin/yast2</annotate> <annotate key="org.freedesktop.policykit.exec.allow_gui">true</annotate> </action> </policyconfig>Save, close – the success can be checked as a regular user with pkexec /usr/sbin/yast2.
- Save the default rights configuration and replace it with the system configuration. Our file will not be overwritten during an upgrade.
mv /etc/polkit-default-privs.local /etc/polkit-default-privs.local.bkup cp /etc/polkit-default-privs.standard /etc/polkit-default-privs.local
The necessary adjustment is to replace auth_admin with auth_self everywhere. You can also do this by hand, but with sed it is more convenient and faster:
sed -i 's/auth_admin/auth_self/g' /etc/polkit-default-privs.local
- To make the authentication via PolicyKit work, create a short shell script that will be called from the menu in the future:
vim /usr/local/sbin/yast2_polkit
- The script looks like this, just add it to the yast2_polkit file:
#!/bin/bash if [ $(which pkexec) ]; then pkexec --disable-internal-agent "/usr/sbin/yast2" "$@" else /usr/sbin/yast2 "$@" fi - Save and close. Finally you make the script executable:
chmod +x /usr/local/sbin/yast2_polkit
- Finally, you create a .desktop file. This will make the modified YaST starter appear directly in the main menu, system-wide for all users. For example, in Xfce it is listed under “Settings”. I have not tested other desktops, but I assume that the starter will end up in a useful place, since it is only a customized copy of the original.
Of course you could also edit the original file for YaST (YaST.desktop) but it will be overwritten during an upgrade. And a copy in /usr/local/share/applications ignores both the application and whisker menus.
So:vim /usr/share/applications/YaST2.desktop
- Insert and save:
[Desktop Entry] X-SuSE-translate=true Type=Application Categories=Settings;System;X-SuSE-Core-System;X-SuSE-ControlCenter-System;X-GNOME-SystemSettings; Name=YaST2 Icon=yast GenericName=Administrator Settings Exec=/usr/local/sbin/yast2_polkit Encoding=UTF-8 Comment=Manage system-wide settings Comment[DE]=Systemweite administrative Einstellungen NoDisplay=false
That’s all. With this, a login as root is no longer necessary or can be done comfortably via sudo su – with your user password. Whether the concept of openSUSE is now worse or better, I don’t want to decide. That is a matter of taste, I think.
What I liked in any case is the clear adherence to standards. This makes finding solutions much easier and faster. Thanks to good documentation and helpful forum posts I was able to finish everything within about an hour – and the great knowledge of PolicyKit!
openSUSE mit sudo – aber komfortabel!
Wer das Handling von Tätigkeiten mit Administratorenrechten (“Root”) wie ich aus der Debian-Welt gewohnt ist, wird mit openSUSE zu Anfang so seine Schwierigkeiten haben. Mit zwei Benutzern geht es noch, da man ja das selbe Passwort sowohl für den User als auch für Root einstellen kann. Aber spätestens bei mehr Nutzerkonten ist damit schon Schluss, außer man gibt das Root-Passwort an alle weiter. Beide Lösungen sind sicher irgendwie praktikabel, aber schön ist das alles nicht. Vor allem weil ja eigentlich sudo installiert wäre – aber nur so halbgar genutzt wird.
Also habe ich mal bei meiner aktuellen openSUSE Tumbleweed daran gemacht, dem System ein vernünftiges sudo-Konzept beizubringen und dieses dann auch für YaST anzuwenden. Das war etwas fieselig herauszufinden, hat am Ende aber gut funktioniert.
Los geht’s!
visudo
Standardmäßig fragt sudo nach dem Root-Passwort. Das ist ziemlich unsinnig, also ändern wir es!
- Im ersten Teil arbeiten wir noch als normaler Benutzer. Die Zeilenangaben können abweichen, je nach Alter der Datei/Systemversion und vorherigen Änderungen daran.
sudo visudo
- Die Parameter in Zeile 43 beginnend mit env_keep = “LANG… ergänzt du am Ende innerhalb der Anführungszeichen mit:
DISPLAY XAUTHORITY
- Die Zeilen 68 und 69 kommentierst du komplett aus, damit nicht mehr das Passwort des “Zielusers” abgefragt wird:
#Defaults targetpw #ALL ALL = (ALL) ALL
- Zusätzlich kommentierst du Zeile 81 ein, löschst also das Kommentarzeichen # weg:
%wheel ALL=(ALL) ALL
- Speichern, schließen und dann deine(n) Benutzer entweder über YaST oder direkt im Terminal der Gruppe “wheel” hinzufügen:
gpasswd -a <dein-username> wheel
Durch Aus- und wieder Einloggen wird die Änderung dann auch übernommen und sudo möchte ab sofort im Terminal immer dein Benutzerpasswort haben.
YaST
Für die grafische Version von YaST wird PolicyKit zur Authentifizierung genutzt, hier ist noch etwas mehr Arbeit nötig. Ab hier arbeitest du als root, wechsle also mit su – den Account.
- Erstelle eine PolicyKit Action für YaST
vim /usr/share/polkit-1/actions/org.opensuse.pkexec.yast2.policy
- Folgenden XML-Block fügst du in die Datei ein. Achte dabei bitte auf Zeilenumbrüche beim Kopieren/Einfügen.
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE policyconfig PUBLIC "-//freedesktop//DTD PolicyKit Policy Configuration 1.0//EN" "http://www.freedesktop.org/software/polkit/policyconfig-1.dtd"> <policyconfig> <action id="org.opensuse.pkexec.yast2"> <message>Authentication is required to run YaST2</message> <icon_name>yast2</icon_name> <defaults> <allow_any>auth_self</allow_any> <allow_inactive>auth_self</allow_inactive> <allow_active>auth_self</allow_active> </defaults> <annotate key="org.freedesktop.policykit.exec.path">/usr/sbin/yast2</annotate> <annotate key="org.freedesktop.policykit.exec.allow_gui">true</annotate> </action> </policyconfig>Speichern, schließen – der Erfolg lässt sich als regulärer Benutzer mit pkexec /usr/sbin/yast2 überprüfen.
- Die vorgegebene Rechte-Konfiguration sicherst du und ersetzt sie durch die System-Konfiguration. Unsere Datei wird bei einem Upgrade übrigens nicht überschrieben.
mv /etc/polkit-default-privs.local /etc/polkit-default-privs.local.bkup cp /etc/polkit-default-privs.standard /etc/polkit-default-privs.local
Die nötige Anpassung ist überall auth_admin durch auth_self zu ersetzen. Du kannst das auch von Hand machen, mit sed geht das aber bequemer und schneller:
sed -i 's/auth_admin/auth_self/g' /etc/polkit-default-privs.local
- Damit die Authentifizierung über PolicyKit auch funktioniert, erstellst du ein kurzes Shellscript das künftig als Umweg aus dem Menü aufgerufen wird:
vim /usr/local/sbin/yast2_polkit
- Das Script sieht wie folgt aus, einfach in die Datei yast2_polkit einfügen:
#!/bin/bash if [ $(which pkexec) ]; then pkexec --disable-internal-agent "/usr/sbin/yast2" "$@" else /usr/sbin/yast2 "$@" fi - Speichern und schließen. Abschließend machst du das Script noch ausführbar:
chmod +x /usr/local/sbin/yast2_polkit
- Als letztes erstellst du eine .desktop Datei. Damit erscheint der modifizierte YaST-Starter direkt im Hauptmenü und das systemweit für alle Benutzer. Beispielsweise wird er bei Xfce unter “Einstellungen” gelistet. Andere Desktops habe ich nicht getestet, gehe aber davon aus dass der Starter an einer sinnvollen Stelle landet da er ja nur eine angepasste Kopie des Originals ist.
Natürlich könntest du auch die originale Datei für YaST (YaST.desktop) bearbeiten aber die wird bei einem Upgrade überschrieben. Und eine Kopie in /usr/local/share/applications ignorieren sowohl das Anwendungs- als auch das Whiskermenü.
Also:
vim /usr/share/applications/YaST2.desktop
- Einfügen und speichern:
[Desktop Entry] X-SuSE-translate=true Type=Application Categories=Settings;System;X-SuSE-Core-System;X-SuSE-ControlCenter-System;X-GNOME-SystemSettings; Name=YaST2 Icon=yast GenericName=Administrator Settings Exec=/usr/local/sbin/yast2_polkit Encoding=UTF-8 Comment=Manage system-wide settings Comment[DE]=Systemweite administrative Einstellungen NoDisplay=false
Das ist alles. Damit ist ein Login als Root nicht mehr nötig bzw. kann bequem über sudo su – mit deinem Benutzerpasswort erfolgen. Ob das Konzept von openSUSE jetzt schlechter oder besser ist, mag ich nicht entscheiden. Das ist Geschmackssache, denke ich.
Was mir auf jeden Fall gut gefallen hat, ist die klare Einhaltung von Standards. Das macht die Suche nach Lösungen deutlich leichter und schneller. Ich konnte dank guter Dokumentation und hilfreichen Forenbeiträgen alles innerhalb von rund einer Stunde fertigstellen – und das große Kenntnisse von PolicyKit!
FlightGear fun
realize "hmm, perhaps I can do the startup thing now", and hit the menu item once again. But instead of running engines, you get fire warnings on all the engines. That does not look good. Confirm fire, extinguish all four engines, and resume looking for airport in range. Trim for best glide. Then number 3 comes up. Then number 4. Number one and you know it will be easy. Number two as you fly over the runway... go around and do normal approach.
Containers 101
This post is written in response to a question by, hacker extraordinaire, Varun on what one should know about Containers as of today. Though the document is mostly generic, some lines of it are India specific, which I have highlighted clearly. Please mention in comments, if there is anything else that should have been covered, or if I have made any mistakes or if you have any opinions.
So, What exactly are Containers ?
Containers are an unit of packaging and deployment, that will guarantee repeatability and isolation. Let us see what each part of that sentence means.Containers are a packaging tool like RPMs or EARs in the sense that they offer you a way to bundle up your binaries (or sources in case of interpreted languages). But instead of merely archiving your sources, Containers provide a way to even deploy your archive, repeatably too.
Anyone who has done packaging, knows, how much of a pain dependency-hell can cause. For example, An application A needs a library L of version 0.1, whereas another application B needs the same library L but of version 0.3 Just to screw up the life of packagers, the versions 0.1 and 0.3 may be conflicting each other and may not co-exist in a system, even in different installation paths. Containerising your application puts each of these applications A and B into their own bundle, with their own library dependencies. However, the real power of containerising is that for each of your application, A and B, they get a view of isolation that they are running in a private environment and so L1 0.1 and 0.3 may never share any runtime data.
One may be reminded about Virtual Machines (VMs) while reading the above text. Even VMs solve the above isolation problem, but they are very heavy. The fundamental difference between a VM and a Container is: a VM virutalizes/abstracts a hardware/operating-system and gives you a machine abstraction, while a Container virtualizes/abstracts an application of your choice. Containers are thus very lightweight and far more approachable.
The Ecosystem
Docker is the most used container technology today. There are other container runtimes such as rkt too. There is an Open Containers Initiative to create standards for container runtimes. All these container runtimes make use of linux kernel features, especially, cgroups to provide process isolation. Microsoft has been making a lot of efforts to support containers natively in the Windows kernel, to support Containers natively as part of their Azure cloud offering for quite some time now.Container Orchestration is a way for deploying different containers on a bunch of machines. While Docker is arguably the champion of container runtimes, Kubernetes is unarguably the King/Queen of container orchestration. Google has been using containers in production, for much long before it became fashionable. In fact the first patch of cgroups support in the linux kernel was submitted to LKML by Google as far back as 2006. Google had/s a large scale cluster management system named Borg which deployed containers (not docker containers) across the humongous google cloud farm. Kubernetes is an open source evolution of Borg, supporting Docker containers natively. Docker-Swarm is an attempt by Docker (the company behind the Docker project) to achieve container orchestration across machines, but there simply is no competition in terms of quality or documentation or feature coverage, compared to Kubernetes (in my limited experience).
Also, in addition to these, There are some poorly implemented, company-specific tools that try to emulate Kubernetes, but these are mostly technical debt and it is wise (imho) for companies to ditch such efforts and move to open projects backed by companies like Google, Red Hat and Microsoft. A distinguished engineer once told me, There is no compression algorithm for experience and there is no need for us to repeat the mistakes made by these companies, decades ago. If you are a startup focussing on solving an user problem, you should focus on your business problem and a container orchestration software should be the last thing that you need to implement.
Kubernetes, though initially a Google project, has now attracted a lot of contributors from a variety of companies such as Red Hat, Microsoft etc. Red Hat have built OpenShift, a platform that provides a lot of useful features such as, Pipelines, Blue-Green deployments, etc. on top of Kubernetes. They even offer a hosted version. Tectonic (on top of Kubernetes) by Core OS is also a big (at least in terms of developer mindshare) player in this ecosystem.
SUSE has come up recently with the Kubic project for containers (even though I have not played with it myself).
Microsoft have hired some high profile names in the container ecosystem for working on the Kubernetes + Azure (Including people like: Brendan Burns, Jess Frazelle, etc.) cloud. Azure is definitely way ahead of Google in India, when it comes to cloud business. Their pricing page is localised for India, while Google does not even support Indian currency yet and charges in USD (leading to jokes like the oil/dollar conspiracy, among the Indian startup ecosystem ;) ). AWS and Azure definitely have a bigger developer mindshare in India than Google Cloud Platform (as of 2017).
The founding team of kubernetes (Xooglers) have started a company named Heptio. While I have no doubts on their engineering prowess, I am skeptical if relying on these companies may be risky for startups in India (lack of same timezone support, etc.). If you are in the west, these options (and others such as rancher) may be interesting.
Kubernetes Basics
In Kubernetes, the unit of deployment is a Pod. A pod is merely a collection of Docker containers which will be deployed together always. For example, if your application is a API server that makes use of a Redis cache, before hitting the database for each request, you create a Pod with two containers, a API server container and a Redis container and you deploy them together.Kubernetes refers to an umbrella of projects that run on a cloud, to manage a cloud. It has various components, such as an API server to interact with the kubernetes system, an agent software named kubelet that runs on each machine in the cloud, a fluentd type of daemon to accumulate logs from various containers and provide a single point of access, a web dashboard, a CLI tool named kubectl to perform various options, etc. In addition to these kubernetes specific components, there are also other services, such as the distributed hashstore etcd (originally from coreos) that you need to setup a basic kubernetes cluster. However, If you are a small company, It'll be wise to make use of GKE or Azure hosting or OpenShift hosting instead of deploying your own kubernetes system managed by your own admins. It is not worth the hassle.
Docker Compose
AWS
Conclusion:
How glib-rs works, part 3: Boxed types
(First part of the series, with index to all the articles)
Now let's get on and see how glib-rs handles boxed types.
Boxed types?
Let's say you are given a sealed cardboard box with something, but you can't know what's inside. You can just pass it on to someone else, or burn it. And since computers are magic duplication machines, you may want to copy the box and its contents... and maybe some day you will get around to opening it.
That's a boxed type. You get a pointer to something, who knows what's inside. You can just pass it on to someone else, burn it — I mean, free it — or since computers are magic, copy the pointer and whatever it points to.
That's exactly the API for boxed types.
typedef gpointer (*GBoxedCopyFunc) (gpointer boxed);
typedef void (*GBoxedFreeFunc) (gpointer boxed);
GType g_boxed_type_register_static (const gchar *name,
GBoxedCopyFunc boxed_copy,
GBoxedFreeFunc boxed_free);
Simple copying, simple freeing
Imagine you have a color...
typedef struct {
guchar r;
guchar g;
guchar b;
} Color;
If you had a pointer to a Color, how would you copy it? Easy:
Color *copy_color (Color *a)
{
Color *b = g_new (Color, 1);
*b = *a;
return b;
}
That is, allocate a new Color, and essentially memcpy() the
contents.
And to free it? A simple g_free() works — there are no internal
things that need to be freed individually.
Complex copying, complex freeing
And if we had a color with a name?
typedef struct {
guchar r;
guchar g;
guchar b;
char *name;
} ColorWithName;
We can't just *a = *b here, as we actually need to copy the string
name. Okay:
ColorWithName *copy_color_with_name (ColorWithName *a)
{
ColorWithName *b = g_new (ColorWithName, 1);
b->r = a->r;
b->g = a->g;
b->b = a->b;
b->name = g_strdup (a->name);
return b;
}
The corresponding free_color_with_name() would g_free(b->name) and then
g_free(b), of course.
Glib-rs and boxed types
Let's look at this by parts. First, a BoxedMemoryManager
trait to define the basic API to manage the
memory of boxed types. This is what defines the copy and free
functions, like above.
pub trait BoxedMemoryManager<T>: 'static {
unsafe fn copy(ptr: *const T) -> *mut T;
unsafe fn free(ptr: *mut T);
}
Second, the actual representation of a Boxed type:
pub struct Boxed<T: 'static, MM: BoxedMemoryManager<T>> {
inner: AnyBox<T>,
_dummy: PhantomData<MM>,
}
This struct is generic over T, the actual type that we will be
wrapping, and MM, something which must implement the
BoxedMemoryManager trait.
Inside, it stores inner, an AnyBox, which we will see shortly.
The _dummy: PhantomData<MM> is a Rust-ism to indicate that although this
struct doesn't actually store a memory manager, it acts as if it does
— it does not concern us here.
The actual representation of boxed data
Let's look at that AnyBox that is stored inside a Boxed:
enum AnyBox<T> {
Native(Box<T>),
ForeignOwned(*mut T),
ForeignBorrowed(*mut T),
}
We have three cases:
-
Native(Box<T>)- this boxed valueTcomes from Rust itself, so we know everything about it! -
ForeignOwned(*mut T)- this boxed valueTcame from the outside, but we own it now. We will have to free it when we are done with it. -
ForeignBorrowed(*mut T)- this boxed valueTcame from the outside, but we are just borrowing it temporarily: we don't want to free it when we are done with it.
For example, if we look at the implementation of the Drop
trait for the Boxed struct, we will indeed see that it
calls the BoxedMemoryManager::free() only if we have a
ForeignOwned value:
impl<T: 'static, MM: BoxedMemoryManager<T>> Drop for Boxed<T, MM> {
fn drop(&mut self) {
unsafe {
if let AnyBox::ForeignOwned(ptr) = self.inner {
MM::free(ptr);
}
}
}
}
If we had a Native(Box<T>) value, it means it came from Rust itself,
and Rust knows how to Drop its own Box<T> (i.e. a chunk of memory
allocated in the heap).
But for external resources, we must tell Rust how to manage them.
Again: in the case where the Rust side owns the reference to the
external boxed data, we have a ForeignOwned and Drop it by
free()ing it; in the case where the Rust side is just borrowing the
data temporarily, we have a ForeignBorrowed and don't touch it when
we are done.
Copying
When do we have to copy a boxed value? For example, when we transfer
from Rust to Glib with full transfer of ownership, i.e. the
to_glib_full() pattern that we saw
before. This is how that trait method is
implemented for Boxed:
impl<'a, T: 'static, MM: BoxedMemoryManager<T>> ToGlibPtr<'a, *const T> for Boxed<T, MM> {
fn to_glib_full(&self) -> *const T {
use self::AnyBox::*;
let ptr = match self.inner {
Native(ref b) => &**b as *const T,
ForeignOwned(p) | ForeignBorrowed(p) => p as *const T,
};
unsafe { MM::copy(ptr) }
}
}
See the MM:copy(ptr) in the last line? That's where the copy
happens. The lines above just get the appropriate pointer to the data
data from the AnyBox and cast it.
There is extra boilerplate in boxed.rs which you can look at; it's
mostly a bunch of trait implementations to copy the boxed data at the
appropriate times (e.g. the FromGlibPtrNone trait), also an
implementation of the Deref trait to get to the contents of a Boxed
/ AnyBox easily, etc. The trait implementations are there just to
make it as convenient as possible to handle Boxed types.
Who implements BoxedMemoryManager?
Up to now, we have seen things like the implementation of Drop for
Boxed, which uses BoxedMemoryManager::free(), and the
implementation of ToGlibPtr which uses ::copy().
But those are just the trait's "abstract" methods, so to speak. What actually implements them?
Glib-rs has a general-purpose macro to wrap Glib types. It can wrap boxed types, shared pointer types, and GObjects. For now we will just look at boxed types.
Glib-rs comes with a macro, glib_wrapper!(), that can be used in
different ways. You can use it to automatically write the boilerplate
for a boxed type like this:
glib_wrapper! {
pub struct Color(Boxed<ffi::Color>);
match fn {
copy => |ptr| ffi::color_copy(mut_override(ptr)),
free => |ptr| ffi::color_free(ptr),
get_type => || ffi::color_get_type(),
}
}
This expands to an internal
glib_boxed_wrapper!() macro that does a few
things. We will only look at particularly interesting bits.
First, the macro creates a newtype around a tuple with 1) the actual
data type you want to box, and 2) a memory manager. In the example
above, the newtype would be called Color, and it would wrap an
ffi:Color (say, a C struct).
pub struct $name(Boxed<$ffi_name, MemoryManager>);
Aha! And that MemoryManager? The macro defines it as a zero-sized
type:
pub struct MemoryManager;
Then it implements the BoxedMemoryManager trait for that
MemoryManager struct:
impl BoxedMemoryManager<$ffi_name> for MemoryManager {
#[inline]
unsafe fn copy($copy_arg: *const $ffi_name) -> *mut $ffi_name {
$copy_expr
}
#[inline]
unsafe fn free($free_arg: *mut $ffi_name) {
$free_expr
}
}
There! This is where the copy/free methods are implemented, based
on the bits of code with which you invoked the macro. In the call to
glib_wrapper!() we had this:
copy => |ptr| ffi::color_copy(mut_override(ptr)),
free => |ptr| ffi::color_free(ptr),
In the impl aboe, the $copy_expr will expand to
ffi::color_copy(mut_override(ptr)) and $free_expr will expand to
ffi::color_free(ptr), which defines our implementation of a memory
manager for our Color boxed type.
Zero-sized what?
Within the macro's definition, let's look again at the definitions of
our boxed type and the memory manager object that actually implements
the BoxedMemoryManager trait. Here is what the macro would expand
to with our Color example:
pub struct Color(Boxed<ffi::Color, MemoryManager>);
pub struct MemoryManager;
impl BoxedMemoryManager<ffi::Color> for MemoryManager {
unsafe fn copy(...) -> *mut ffi::Color { ... }
unsafe fn free(...) { ... }
}
Here, MemoryManager is a zero-sized type. This means it doesn't
take up any space in the Color tuple! When a Color is allocated
in the heap, it is really as if it contained an ffi::Color (the
C struct we are wrapping) and nothing else.
All the knowledge about how to copy/free ffi::Color lives only in
the compiler thanks to the trait implementation. When the compiler
expands all the macros and monomorphizes all the generic functions,
the calls to ffi::color_copy() and ffi::color_free() will be
inlined at the appropriate spots. There is no need to have
auxiliary structures taking up space in the heap, just to store
function pointers to the copy/free functions, or anything like that.
Next up
You may have seen that our example call to glib_wrapper!() also
passed in a ffi::color_get_type() function. We haven't talked about
how glib-rs wraps Glib's GType, GValue, and all of that. We are
getting closer and closer to being able to wrap GObject.
Stay tuned!
Ceph Meetup Berlin
I will give a presentation at the next Ceph Meetup in Berlin on the 18th of September. It will be about a exciting project we work on, at Deutsche Telekom, since a while. The goal of this open source project called librmb is to store emails directly in Ceph RADOS instead of using e.g. a NAS system.
Initial posts about librsvg's C to Rust conversion
The initial articles about librsvg's conversion to Rust are in my old blog, so they may be a bit hard to find from this new blog. Here is a list of those posts, just so they are easier to find:
- Librsvg gets Rusty
- Porting a few C functions to Rust
- Bézier curves, markers, and SVG's concept of directionality
- Refactoring C to make Rustification easier
- Exposing Rust objects to C code
- Debugging Rust code inside a C library
- Reproducible font rendering for librsvg's tests
- Algebraic data types in Rust, and basic parsing
- How librsvg exports reference-counted objects from Rust to C
- Griping about parsers and shitty specifications
- Porting librsvg's tree of nodes to Rust
- gboolean is not Rust bool
Within this new blog, you can look for articles with the librsvg tag.