<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
    <title>kttnr</title>
    <link rel="self" type="application/atom+xml" href="https://kttnr.net/atom.xml"/>
    <link rel="alternate" type="text/html" href="https://kttnr.net"/>
    <generator uri="https://www.getzola.org/">Zola</generator>
    <updated>2026-05-25T00:00:00+00:00</updated>
    <id>https://kttnr.net/atom.xml</id>
    <entry xml:lang="en">
        <title>page down behavior</title>
        <published>2026-05-25T00:00:00+00:00</published>
        <updated>2026-05-25T00:00:00+00:00</updated>
        
        <author>
          <name>
            
              Unknown
            
          </name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://kttnr.net/blog/page-down-behavior/"/>
        <id>https://kttnr.net/blog/page-down-behavior/</id>
        
        <content type="html" xml:base="https://kttnr.net/blog/page-down-behavior/">&lt;p&gt;Page down could work better on the last page of content in web browsers in similar applications.&lt;&#x2F;p&gt;
&lt;p&gt;When reading content that doesn&#x27;t fit on one page, I use the page down key to get to the next visual page. Page down scrolls down so that what was previously at the bottom of the screen goes to the top. I immediately know where to place my eyes to continue reading.&lt;&#x2F;p&gt;
&lt;p&gt;This breaks near the end of the content. If there is only half a screen worth of content left, then page down reaches the end of the article and the next line of content ends up in the middle of the screen instead of the top. I don&#x27;t know that this is the last page or how much content is left. I don&#x27;t know where to place my eyes. I have to scan the whole page top to bottom to find where I left off. This is annoying.&lt;&#x2F;p&gt;
&lt;p&gt;I wish page down had the same behavior on the last page as it does on the previous pages. Applications could implement this by adding an extra screen&#x27;s worth of space at the end of the content. This would also allow manually scrolling there. I sometimes want this to place something higher on the screen without moving the application window.&lt;&#x2F;p&gt;
&lt;p&gt;The new behavior should not apply to the end key, which scrolls to the end of a page. That should continue to work as before, placing the end of the content at the bottom of the screen.&lt;&#x2F;p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>sandboxing Wayland applications</title>
        <published>2026-05-15T00:00:00+00:00</published>
        <updated>2026-05-15T00:00:00+00:00</updated>
        
        <author>
          <name>
            
              Unknown
            
          </name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://kttnr.net/blog/sandboxing-wayland-applications/"/>
        <id>https://kttnr.net/blog/sandboxing-wayland-applications/</id>
        
        <content type="html" xml:base="https://kttnr.net/blog/sandboxing-wayland-applications/">&lt;p&gt;I like sandboxing Linux applications. Tools like &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;containers&#x2F;bubblewrap&quot;&gt;bubblewrap&lt;&#x2F;a&gt; and &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;igo95862&#x2F;bubblejail&quot;&gt;Bubblejail&lt;&#x2F;a&gt; make this easy, building on &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Linux_namespaces&quot;&gt;Linux namespaces&lt;&#x2F;a&gt;.&lt;&#x2F;p&gt;
&lt;p&gt;This works well for command line applications, but sandboxing graphical applications is harder. A graphical Wayland application needs to access the Wayland socket to handle its windows. The Wayland socket gives access to sensitive data like screen contents. You might want to restrict this. With the &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;wayland.emersion.fr&#x2F;protocol&#x2F;security-context-v1.html&quot;&gt;security_context&lt;&#x2F;a&gt; protocol, you can.&lt;&#x2F;p&gt;
&lt;p&gt;Instead of mounting the normal Wayland socket into the sandbox, mount a socket that has the security context applied. You can create such a socket with &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;git.sr.ht&#x2F;~whynothugo&#x2F;way-secure&quot;&gt;way-secure&lt;&#x2F;a&gt;. This makes Wayland applications work safely.&lt;&#x2F;p&gt;
&lt;p&gt;I mainly wanted to write about the Wayland socket in this post because there isn&#x27;t much awareness of the bubblewrap plus way-secure approach. The Wayland socket is not everything. The application can show a window, but other desktop application features are missing. This is the same problem that &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;flatpak.org&#x2F;&quot;&gt;Flatpak&lt;&#x2F;a&gt; solves with tools like &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;flatpak.github.io&#x2F;xdg-desktop-portal&#x2F;&quot;&gt;XDG Desktop Portal&lt;&#x2F;a&gt;. See what Flatpak permissions are available &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;docs.flatpak.org&#x2F;en&#x2F;latest&#x2F;sandbox-permissions.html&quot;&gt;here&lt;&#x2F;a&gt;. You can reuse some of the Flatpak tools and approaches. Bubblejail does this to some degree but needs more polish.&lt;&#x2F;p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Firefox tracking protection Google whitelist</title>
        <published>2025-09-29T00:00:00+00:00</published>
        <updated>2025-09-29T00:00:00+00:00</updated>
        
        <author>
          <name>
            
              Unknown
            
          </name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://kttnr.net/blog/firefox-tracking-protection-google-whitelist/"/>
        <id>https://kttnr.net/blog/firefox-tracking-protection-google-whitelist/</id>
        
        <content type="html" xml:base="https://kttnr.net/blog/firefox-tracking-protection-google-whitelist/">&lt;p&gt;Some Firefox configuration strings give the incorrect impression that Firefox excludes Google from tracking protection. These strings and their explanation are difficult to find through internet search. This article is that explanation.&lt;&#x2F;p&gt;
&lt;p&gt;Firefox has built in tracking protection with configuration in &lt;code&gt;about:config&lt;&#x2F;code&gt;. Some of the config keys contain &quot;whitelist&quot;:&lt;&#x2F;p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;urlclassifier.features.fingerprinting.annotate.whitelistTables&lt;&#x2F;code&gt;&lt;&#x2F;li&gt;
&lt;li&gt;&lt;code&gt;urlclassifier.features.fingerprinting.whitelistTables&lt;&#x2F;code&gt;&lt;&#x2F;li&gt;
&lt;li&gt;&lt;code&gt;urlclassifier.features.socialtracking.annotate.whitelistTables&lt;&#x2F;code&gt;&lt;&#x2F;li&gt;
&lt;li&gt;&lt;code&gt;urlclassifier.features.socialtracking.whitelistTables&lt;&#x2F;code&gt;&lt;&#x2F;li&gt;
&lt;li&gt;&lt;code&gt;urlclassifier.trackingAnnotationWhitelistTable&lt;&#x2F;code&gt;&lt;&#x2F;li&gt;
&lt;li&gt;&lt;code&gt;urlclassifier.trackingWhitelistTable&lt;&#x2F;code&gt;&lt;&#x2F;li&gt;
&lt;&#x2F;ul&gt;
&lt;p&gt;The keys store a list with these elements:&lt;&#x2F;p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;mozstd-trackwhite-digest256&lt;&#x2F;code&gt;&lt;&#x2F;li&gt;
&lt;li&gt;&lt;code&gt;google-trackwhite-digest256&lt;&#x2F;code&gt;&lt;&#x2F;li&gt;
&lt;&#x2F;ul&gt;
&lt;p&gt;This gives the incorrect impression that Firefox excludes Google from tracking protection.&lt;&#x2F;p&gt;
&lt;p&gt;Firefox loads the tracking protection database at runtime. One part of the database stores which URLs to block. Another part stores a list of exceptions when not to block. This is the &quot;whitelist&quot;. It exists because it doesn&#x27;t make sense to block a tracking domain in all circumstances. You want to block tracking domains embedded on other domains. For example, on &lt;code&gt;my-personal-blog.com&lt;&#x2F;code&gt; you want to block &lt;code&gt;tracker.evilcorp.com&lt;&#x2F;code&gt;. But on the main &lt;code&gt;evilcorp.com&lt;&#x2F;code&gt; it doesn&#x27;t make sense to block the tracking subdomain. The same company controls both domains and can already track you through the domain main. The tracking subdomain doesn&#x27;t add any extra tracking capabilities. Additionally, blocking tracking subdomains often breaks the main website.&lt;&#x2F;p&gt;
&lt;p&gt;Firefox used to have only one whitelist called &lt;code&gt;mozstd-trackwhite&lt;&#x2F;code&gt;. At some point they added many Google domains to it. This made the it grow in size significantly which triggered a bug in the code that loads it. As a workaround they separated the new domains into another list called &lt;code&gt;google-trackwhite&lt;&#x2F;code&gt;.&lt;&#x2F;p&gt;
&lt;p&gt;The list is not made by Google. The list does not favor Google. The list just contains a subset of the changes intended for the main list. In the future Firefox might merge the lists again. The whitelist configuration is not evidence of Firefox favoring Google.&lt;&#x2F;p&gt;
&lt;p&gt;Sources:&lt;&#x2F;p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;linuxreviews.org&#x2F;Mozilla_Is_Rolling_Out_Redirect_Tracking_Protection_In_Firefox_In_A_Somewhat_Concerning_Fashion&quot;&gt;example of the misconception&lt;&#x2F;a&gt;&lt;&#x2F;li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;bugzilla.mozilla.org&#x2F;show_bug.cgi?id=1602348&quot;&gt;Firefox issue for separating the lists&lt;&#x2F;a&gt;&lt;&#x2F;li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;mozilla-services&#x2F;shavar-prod-lists&#x2F;pull&#x2F;79#issuecomment-547186843&quot;&gt;Github comment explaining the list growing too large&lt;&#x2F;a&gt;&lt;&#x2F;li&gt;
&lt;&#x2F;ul&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Rust HashMap limitations</title>
        <published>2024-12-16T00:00:00+00:00</published>
        <updated>2024-12-16T00:00:00+00:00</updated>
        
        <author>
          <name>
            
              Unknown
            
          </name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://kttnr.net/blog/rust-hashmap-limitations/"/>
        <id>https://kttnr.net/blog/rust-hashmap-limitations/</id>
        
        <content type="html" xml:base="https://kttnr.net/blog/rust-hashmap-limitations/">&lt;p&gt;This post gives examples of API limitations in the Rust standard library&#x27;s &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;std&#x2F;collections&#x2F;struct.HashMap.html&quot;&gt;&lt;code&gt;HashMap&lt;&#x2F;code&gt;&lt;&#x2F;a&gt;. The limitations make some code slower than necessary. The limitations are on the API level. You don&#x27;t need to change much implementation code to fix them but you need to change stable standard library APIs.&lt;&#x2F;p&gt;
&lt;h2 id=&quot;entry&quot;&gt;entry&lt;&#x2F;h2&gt;
&lt;p&gt;HashMap has an &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;std&#x2F;collections&#x2F;struct.HashMap.html#method.entry&quot;&gt;entry API&lt;&#x2F;a&gt;. Its purpose is to allow you to operate on a key in the map multiple times while looking up the key only once. Without this API, you would need to look up the key for each operation, which is slow.&lt;&#x2F;p&gt;
&lt;p&gt;Here is an example of an operation without the entry API:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;fn insert_or_increment(key: String, hashmap: &amp;amp;mut HashMap&amp;lt;String, u32&amp;gt;) {
    if let Some(stored_value) = hashmap.get_mut(&amp;amp;key) {
        *stored_value += 1;
    } else {
        hashmap.insert(key, 1);
    }
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;This operation looks up the key twice. First in &lt;code&gt;get_mut&lt;&#x2F;code&gt;, then in &lt;code&gt;insert&lt;&#x2F;code&gt;.&lt;&#x2F;p&gt;
&lt;p&gt;Here is the equivalent code with the entry API:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;fn insert_or_increment(key: String, hashmap: &amp;amp;mut HashMap&amp;lt;String, u32&amp;gt;) {
    hashmap
        .entry(key)
        .and_modify(|value| *value += 1)
        .or_insert(1);
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;This operation looks up the key once in &lt;code&gt;entry&lt;&#x2F;code&gt;.&lt;&#x2F;p&gt;
&lt;p&gt;Unfortunately, the entry API has a limitation. It takes the key by value. It does this because when you insert a new entry, the hash table needs to take ownership of the key. However, you might not always decide to insert a new entry after seeing the existing entry. In the example above we only insert if there is no existing entry. This matters when you have a reference to the key and turning it into an owned value is expensive.&lt;&#x2F;p&gt;
&lt;p&gt;Consider this modification of the previous example. We now take the key as a string reference rather than a string value:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;fn insert_or_increment(key: &amp;amp;str, hashmap: &amp;amp;mut HashMap&amp;lt;String, u32&amp;gt;) {
    hashmap
        .entry(key.to_owned())
        .and_modify(|value| *value += 1)
        .or_insert(1);
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;We had to change &lt;code&gt;entry(key)&lt;&#x2F;code&gt; to &lt;code&gt;entry(key.to_owned())&lt;&#x2F;code&gt;, cloning the string. This is expensive. It would be better if we only cloned the string in the &lt;code&gt;or_insert&lt;&#x2F;code&gt; case. We can accomplish by not using the entry API like in this modification of the first example.&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;fn insert_or_increment(key: &amp;amp;str, hashmap: &amp;amp;mut HashMap&amp;lt;String, u32&amp;gt;) {
    if let Some(stored_value) = hashmap.get_mut(key) {
        *stored_value += 1;
    } else {
        hashmap.insert(key.to_owned(), 1);
    }
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;But now we cannot get the benefit of the entry API. We have to pick between two inefficiencies.&lt;&#x2F;p&gt;
&lt;p&gt;This problem could be avoided if the entry API supported taking the key by reference (more accurately: by borrow) or by &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;std&#x2F;borrow&#x2F;enum.Cow.html&quot;&gt;&lt;code&gt;Cow&lt;&#x2F;code&gt;&lt;&#x2F;a&gt;. The entry API could then internally use &lt;code&gt;to_owned&lt;&#x2F;code&gt; when necessary.&lt;&#x2F;p&gt;
&lt;p&gt;The custom hash table implementation in the hashbrown crate &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;docs.rs&#x2F;hashbrown&#x2F;latest&#x2F;hashbrown&#x2F;struct.HashMap.html#method.entry_ref&quot;&gt;implements&lt;&#x2F;a&gt; this improvement. &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;internals.rust-lang.org&#x2F;t&#x2F;head-desking-on-entry-api-4-0&#x2F;2156&quot;&gt;Here&lt;&#x2F;a&gt; is a post from 2015 by Gankra that goes into more detail on why the standard library did not do this.&lt;&#x2F;p&gt;
&lt;h2 id=&quot;borrow&quot;&gt;Borrow&lt;&#x2F;h2&gt;
&lt;p&gt;The various HashMap &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;std&#x2F;collections&#x2F;struct.HashMap.html#method.contains_key&quot;&gt;functions&lt;&#x2F;a&gt; that look up keys do not take a reference to the key type. Their signature looks like this:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;pub fn contains_key&amp;lt;Q&amp;gt;(&amp;amp;self, k: &amp;amp;Q) -&amp;gt; bool
where
    K: Borrow&amp;lt;Q&amp;gt;,
    Q: Hash + Eq + ?Sized,
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;They take a type Q, which the hash table&#x27;s key type can be borrowed as. This happens through the &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;std&#x2F;borrow&#x2F;trait.Borrow.html&quot;&gt;borrow&lt;&#x2F;a&gt; trait. This makes keys more flexible and allows code to be more efficient. For example, &lt;code&gt;String&lt;&#x2F;code&gt; as the key type still allows look up by &lt;code&gt;&amp;amp;str&lt;&#x2F;code&gt; in addition of &lt;code&gt;&amp;amp;String&lt;&#x2F;code&gt;. This is good because it is expensive to turn &lt;code&gt;&amp;amp;str&lt;&#x2F;code&gt; into &lt;code&gt;&amp;amp;String&lt;&#x2F;code&gt;. You can only do this by cloning the string. Generic keys through the borrow trait allow us to work with &lt;code&gt;&amp;amp;str&lt;&#x2F;code&gt; directly, omitting the clone.&lt;&#x2F;p&gt;
&lt;p&gt;Unfortunately the borrow API has a limitation. It is impossible to implement in some cases.&lt;&#x2F;p&gt;
&lt;p&gt;Consider the following example, which uses a custom key type:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;#[derive(Eq, PartialEq, Hash)]
struct Key {
    a: String,
    b: String,
}

type MyHashMap = HashMap&amp;lt;Key, ()&amp;gt;;

fn contains_key(key: &amp;amp;Key, hashmap: &amp;amp;MyHashMap) -&amp;gt; bool {
    hashmap.contains_key(key)
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Now consider a function that takes two key strings individually by reference, instead of the whole key struct by reference:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;fn contains_key(key_a: &amp;amp;str, key_b: &amp;amp;str, hashmap: &amp;amp;MyHashMap) -&amp;gt; bool {
    todo!()
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;How do we implement the function body? We want to avoid expensive clones of the input strings. It seems like this is what the borrow trait is made for. Let&#x27;s create a wrapper struct that represents a custom key reference. The struct functions &lt;code&gt;&amp;amp;str&lt;&#x2F;code&gt; instead of &lt;code&gt;&amp;amp;String&lt;&#x2F;code&gt;.&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;#[derive(Eq, PartialEq, Hash)]
struct KeyRef&amp;lt;&amp;#39;a&amp;gt; {
    a: &amp;amp;&amp;#39;a str,
    b: &amp;amp;&amp;#39;a str,
}

impl&amp;lt;&amp;#39;a&amp;gt; Borrow&amp;lt;KeyRef&amp;lt;&amp;#39;a&amp;gt;&amp;gt; for Key {
    fn borrow(&amp;amp;self) -&amp;gt; &amp;amp;KeyRef&amp;lt;&amp;#39;a&amp;gt; {
        &amp;amp;KeyRef {
            a: &amp;amp;self.a,
            b: &amp;amp;self.b,
        }
    }
}

fn contains_key(key_a: &amp;amp;str, key_b: &amp;amp;str, hashmap: &amp;amp;MyHashMap) -&amp;gt; bool {
    let key_ref = KeyRef { a: key_a, b: key_b };
    hashmap.contains_key(&amp;amp;key_ref)
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;This does not compile. In the borrow function we attempt to return a reference to a local value. This is a lifetime error. The local value would go out of scope when the function returns, making the reference invalid. We cannot fix this. The borrow trait requires returning a reference. We cannot return a value. This is fine for &lt;code&gt;String&lt;&#x2F;code&gt; to &lt;code&gt;&amp;amp;str&lt;&#x2F;code&gt; or &lt;code&gt;Vec&amp;lt;u8&amp;gt;&lt;&#x2F;code&gt; to &lt;code&gt;&amp;amp;[u8]&lt;&#x2F;code&gt;, but it does not work for our key type.&lt;&#x2F;p&gt;
&lt;p&gt;This problem could be avoided by changing the borrow trait or introducing a new trait for this purpose.&lt;&#x2F;p&gt;
&lt;p&gt;(In the specific example above, we could workaround this limitation by changing our key type to store &lt;code&gt;Cow&amp;lt;str&amp;gt;&lt;&#x2F;code&gt; instead of &lt;code&gt;String&lt;&#x2F;code&gt;. This is worse than the &lt;code&gt;KeyRef&lt;&#x2F;code&gt; solution because it is slower because now all of our keys are enums. A more efficient, more boilerplate workaround is using &lt;code&gt;&amp;amp;dyn&lt;&#x2F;code&gt; trait objects as &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;sunshowers-code&#x2F;borrow-complex-key-example&quot;&gt;documented&lt;&#x2F;a&gt; by user sunshowers.)&lt;&#x2F;p&gt;
&lt;p&gt;The custom hash table implementation in the hashbrown crate implements this improvement. Hashbrown uses a better designed &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;docs.rs&#x2F;hashbrown&#x2F;0.15.2&#x2F;hashbrown&#x2F;trait.Equivalent.html&quot;&gt;custom trait&lt;&#x2F;a&gt; instead of the standard borrow trait.&lt;&#x2F;p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>fast float to integer conversions</title>
        <published>2024-11-10T00:00:00+00:00</published>
        <updated>2024-11-10T00:00:00+00:00</updated>
        
        <author>
          <name>
            
              Unknown
            
          </name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://kttnr.net/blog/fast-float-to-integer-conversions/"/>
        <id>https://kttnr.net/blog/fast-float-to-integer-conversions/</id>
        
        <content type="html" xml:base="https://kttnr.net/blog/fast-float-to-integer-conversions/">&lt;p&gt;In Rust, the standard way of converting floating point values to integers is with the &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;reference&#x2F;expressions&#x2F;operator-expr.html#type-cast-expressions&quot;&gt;&lt;code&gt;as&lt;&#x2F;code&gt; operator&lt;&#x2F;a&gt;. This conversion has various guarantees as listed in the reference. One of them is that it saturates: Input values out of range of the output type convert to the minimal&#x2F;maximal value of the output type.&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;assert_eq!(300f32 as u8, 255);
assert_eq!(-5f32 as u8, 0);
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Saturation comes with a downside. It is slower than not saturating. In C&#x2F;C++ this kind of cast is &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;e00E&#x2F;cpp-clamp-cast&quot;&gt;undefined behavior&lt;&#x2F;a&gt;. Rust&#x27;s casts are never undefined behavior (good), but they are slower than C&#x2F;C++.&lt;&#x2F;p&gt;
&lt;p&gt;On many &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;nightly&#x2F;rustc&#x2F;platform-support.html&quot;&gt;hardware targets&lt;&#x2F;a&gt; a float to integer conversion can be done in one instruction. For example &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;www.felixcloutier.com&#x2F;x86&#x2F;cvttss2si&quot;&gt;&lt;code&gt;CVTTSS2SI&lt;&#x2F;code&gt;&lt;&#x2F;a&gt; on x86_84+SSE. This is the minimum amount of work we need to do to perform the conversion. This is what the C&#x2F;C++ cast compiles to. Rust needs more instructions because of the additional guarantees.&lt;&#x2F;p&gt;
&lt;p&gt;Sometimes you want faster conversions and don&#x27;t need saturation. You might already know that your inputs are in range or you might not care about the result for out of range inputs. My new &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;e00E&#x2F;fast-float-to-integer&quot;&gt;fast-float-to-integer crate&lt;&#x2F;a&gt; provides this. If in range, then the conversion functions work like the standard &lt;code&gt;as&lt;&#x2F;code&gt; operator conversion. If not in range, then you get an unspecified value. We get the speed of the C&#x2F;C++ version without the undefined behavior.&lt;&#x2F;p&gt;
&lt;p&gt;You never get undefined behavior but you can get unspecified behavior. In the unspecified case, you get an arbitrary value. The function returns and you get a valid value of the output type, but there is no guarantee what that value is. On the assembly level the result is specified but we cannot guarantee this in the crate because the behavior varies between platforms.&lt;&#x2F;p&gt;
&lt;h2 id=&quot;assembly&quot;&gt;assembly&lt;&#x2F;h2&gt;
&lt;p&gt;We keep track of the generated assembly in the repository and enforce on CI that it is up to date. Here is a typical assembly comparison on x86_64+SSE.&lt;&#x2F;p&gt;
&lt;p&gt;standard:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;asm&quot;&gt;f32_to_i64:
    cvttss2si rax, xmm0
    ucomiss xmm0, dword ptr [rip + .L_0]
    movabs rcx, 9223372036854775807
    cmovbe rcx, rax
    xor eax, eax
    ucomiss xmm0, xmm0
    cmovnp rax, rcx
    ret
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;fast:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;asm&quot;&gt;f32_to_i64:
    cvttss2si rax, xmm0
    ret
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;As expected, we do the conversion in one instruction. The extra assembly in the standard version performs the saturation.&lt;&#x2F;p&gt;
&lt;p&gt;A more complicated case is f32 to u64. We cannot handle this in one instruction, because the cvttss2si instruction outputs an i64. It cannot handle the values between &lt;code&gt;i64::MAX&lt;&#x2F;code&gt; and &lt;code&gt;u64::MAX&lt;&#x2F;code&gt;. (AVX512 could do this, but the intrinsics are not stable and it might not be faster.) With some clever optimization, we can still do better than the standard version. I copied the following Rust code from the &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;e00E&#x2F;fast-float-to-integer&#x2F;blob&#x2F;5ba207a2188031abcf285f8cbd7ef85f7a1f5b8f&#x2F;src&#x2F;target_x86_64_sse.rs#L40&quot;&gt;repository&lt;&#x2F;a&gt;.&lt;&#x2F;p&gt;
&lt;p&gt;standard:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;asm&quot;&gt;f32_to_u64:
    cvttss2si rax, xmm0
    mov rcx, rax
    sar rcx, 63
    movaps xmm1, xmm0
    subss xmm1, dword ptr [rip + .L_0]
    cvttss2si rdx, xmm1
    and rdx, rcx
    or rdx, rax
    xor ecx, ecx
    xorps xmm1, xmm1
    ucomiss xmm0, xmm1
    cmovae rcx, rdx
    ucomiss xmm0, dword ptr [rip + .L_1]
    mov rax, -1
    cmovbe rax, rcx
    ret
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;fast:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;asm&quot;&gt;f32_to_u64:
    cvttss2si rcx, xmm0
    addss xmm0, dword ptr [rip + .L_0]
    cvttss2si rdx, xmm0
    mov rax, rcx
    sar rax, 63
    and rax, rdx
    or rax, rcx
    ret
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;First, let us consider a relatively naive branchful solution:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;#[inline(always)]
fn _f32_to_u64_branchful(float: f32) -&amp;gt; u64 {
    const THRESHOLD_FLOAT: f32 = power_of_two_f32(63);
    const THRESHOLD_INTEGER: u64 = 2u64.pow(63);

    let in_range = float &amp;lt;= THRESHOLD_FLOAT;
    if in_range {
        f32_to_i64(float) as u64
    } else {
        &#x2F;&#x2F; Subtract the threshold from the float. The result is &amp;gt;= 0 because the input is larger
        &#x2F;&#x2F; than the subtrahend. The result is &amp;lt;= i64::MAX because `u64::MAX - i64::MAX == i64::MAX`.
        let in_range_float = float - THRESHOLD_FLOAT;
        let integer = f32_to_i64(in_range_float) as u64;
        &#x2F;&#x2F; Overflow is benign because it can only occur for invalid inputs.
        integer.overflowing_add(THRESHOLD_INTEGER).0
    }
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;We can turn this into a less naive branchless solution:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;#[inline(always)]
fn f32_to_u64_branchless(float: f32) -&amp;gt; u64 {
    const THRESHOLD: f32 = power_of_two_f32(63);

    let integer1 = f32_to_i64(float);
    let integer2 = f32_to_i64(float - THRESHOLD);
    &#x2F;&#x2F; If the input is larger than i64::MAX, then integer1 is i64::MIN. This value has 1 as the
    &#x2F;&#x2F; leftmost bit and 0 as the remaining bits. Right shift on signed values is arithmetic, not
    &#x2F;&#x2F; logical [1]. We end up with all 0 (in range) or all 1 (out of range).
    let too_large = integer1 &amp;gt;&amp;gt; 63;
    &#x2F;&#x2F; # If the input is not too large:
    &#x2F;&#x2F;
    &#x2F;&#x2F; Integer1 has the correct value. The mask is all 0, which makes the Or result in integer1.
    &#x2F;&#x2F;
    &#x2F;&#x2F; # If the input is too large:
    &#x2F;&#x2F;
    &#x2F;&#x2F; Integer1 is i64::MIN and the mask is all 1. The Or results in `i64::MIN | integer2`.
    &#x2F;&#x2F; integer2 has the correct result minus 2**63. This is the correct result without the
    &#x2F;&#x2F; leftmost bit. The Or adds the missing leftmost bit back.
    (integer1 | (integer2 &amp;amp; too_large)) as u64
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;This compiles to the assembly above.&lt;&#x2F;p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>missed Rust optimizations</title>
        <published>2023-07-14T00:00:00+00:00</published>
        <updated>2023-07-14T00:00:00+00:00</updated>
        
        <author>
          <name>
            
              Unknown
            
          </name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://kttnr.net/blog/missed-rust-optimizations/"/>
        <id>https://kttnr.net/blog/missed-rust-optimizations/</id>
        
        <content type="html" xml:base="https://kttnr.net/blog/missed-rust-optimizations/">&lt;p&gt;I like inspecting the optimized assembly instructions of small self contained parts of my Rust programs. I sometimes find code that doesn&#x27;t optimize well. This is especially interesting when it is possible to rewrite the code in a semantically equivalent way which optimizes better. I want to give some examples of this as evidence that it can be worth it to try to improve generated assembly. Compilers are neither perfect nor always better than humans.&lt;&#x2F;p&gt;
&lt;p&gt;The presented examples all come from real code that I wrote. They are not contrived. You can find more examples in the Rust issue tracker with the &quot;slow&quot; &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;rust-lang&#x2F;rust&#x2F;issues?q=is%3Aissue+label%3AI-slow&quot;&gt;label&lt;&#x2F;a&gt;.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;pacak&#x2F;cargo-show-asm&quot;&gt;cargo-show-asm&lt;&#x2F;a&gt; and &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;godbolt.org&#x2F;&quot;&gt;Compiler Explorer&lt;&#x2F;a&gt; work well for looking at Rust compiler output.&lt;&#x2F;p&gt;
&lt;h2 id=&quot;example-one&quot;&gt;example one&lt;&#x2F;h2&gt;
&lt;p&gt;&lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;godbolt.org&#x2F;z&#x2F;4YTjWeWoa&quot;&gt;Compiler Explorer&lt;&#x2F;a&gt;, &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;rust-lang&#x2F;rust&#x2F;issues&#x2F;85841&quot;&gt;Rust issue&lt;&#x2F;a&gt;&lt;&#x2F;p&gt;
&lt;p&gt;We have a simple enum on which we want to implement an iterator style &lt;code&gt;next&lt;&#x2F;code&gt; function.&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;#[derive(Clone, Copy)]
enum E {
    E0,
    E1,
    E2,
    E3,
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;You might implement &lt;code&gt;next&lt;&#x2F;code&gt; like this:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;fn next_v0(e: E) -&amp;gt; Option&amp;lt;E&amp;gt; {
    Some(match e {
        E::E0 =&amp;gt; E::E1,
        E::E1 =&amp;gt; E::E2,
        E::E2 =&amp;gt; E::E3,
        E::E3 =&amp;gt; return None,
    })
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Which produces this assembly:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;asm&quot;&gt;example::next_v0:
        mov     al, 4
        mov     cl, 1
        movzx   edx, dil
        lea     rsi, [rip + .LJTI0_0]
        movsxd  rdx, dword ptr [rsi + 4*rdx]
        add     rdx, rsi
        jmp     rdx
.LBB0_2:
        mov     cl, 2
        jmp     .LBB0_3
.LBB0_1:
        mov     cl, 3
.LBB0_3:
        mov     eax, ecx
.LBB0_4:
        ret
.LJTI0_0:
        .long   .LBB0_3-.LJTI0_0
        .long   .LBB0_2-.LJTI0_0
        .long   .LBB0_1-.LJTI0_0
        .long   .LBB0_4-.LJTI0_0
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;The match expression turns into a jump table with 4 branches. You would expect this assembly, if we did some arbitrary operation in each match case, that isn&#x27;t related to the other cases. However, If you are familiar with how Rust represents enums and options, you might realize that this is not optimal.&lt;&#x2F;p&gt;
&lt;p&gt;The enum is 1 byte large. The variants are represented as 0, 1, 2, 3. This representation is not guaranteed (unless you use the &lt;code&gt;repr&lt;&#x2F;code&gt; attribute) but it is how enums are represented today. The enum only uses 4 out of the 256 possible values. To save space in &lt;code&gt;Option&lt;&#x2F;code&gt;s the Rust compiler performs &quot;niche optimization&quot;. An option needs one more pattern to represent the empty case &lt;code&gt;None&lt;&#x2F;code&gt;. If the inner type has a a free variant, the niche, then it can be used for that. In fact, the representation of &lt;code&gt;Option::&amp;lt;E&amp;gt;::None&lt;&#x2F;code&gt; is 4. To implement &lt;code&gt;next&lt;&#x2F;code&gt; we just need to increment the byte.&lt;&#x2F;p&gt;
&lt;p&gt;Unfortunately the compiler does not realize this unless we rewrite the function like this:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;fn next_v1(e: E) -&amp;gt; Option&amp;lt;E&amp;gt; {
    match e {
        E::E0 =&amp;gt; Some(E::E1),
        E::E1 =&amp;gt; Some(E::E2),
        E::E2 =&amp;gt; Some(E::E3),
        E::E3 =&amp;gt; None,
    }
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Which produces this assembly:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;asm&quot;&gt;example::next_v1:
        lea     eax, [rdi + 1]
        ret
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;This is better. There are only two instructions and no branches.&lt;&#x2F;p&gt;
&lt;h2 id=&quot;example-two&quot;&gt;example two&lt;&#x2F;h2&gt;
&lt;p&gt;&lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;godbolt.org&#x2F;z&#x2F;6Pv4bxs1b&quot;&gt;Compiler Explorer&lt;&#x2F;a&gt;, &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;rust-lang&#x2F;rust&#x2F;issues&#x2F;113691&quot;&gt;Rust issue&lt;&#x2F;a&gt;&lt;&#x2F;p&gt;
&lt;p&gt;We have an array of 5 boolean values and want to return whether all of them are true.&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;pub fn iter_all(a: [bool; 5]) -&amp;gt; bool {
    a.iter().all(|s| *s)
}

pub fn iter_fold(a: [bool; 5]) -&amp;gt; bool {
    a.iter().fold(true, |acc, i| acc &amp;amp; i)
}

pub fn manual_loop(a: [bool; 5]) -&amp;gt; bool {
    let mut b = true;
    for a in a {
        b &amp;amp;= a;
    }
    b
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;&lt;code&gt;iter_all&lt;&#x2F;code&gt;, &lt;code&gt;iter_fold&lt;&#x2F;code&gt;, &lt;code&gt;manual_loop&lt;&#x2F;code&gt; produce the same assembly:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;asm&quot;&gt;example::iter_all:
        movabs  rax, 1099511627775
        and     rax, rdi
        test    dil, dil
        setne   cl
        test    edi, 65280
        setne   dl
        and     dl, cl
        test    edi, 16711680
        setne   cl
        test    edi, -16777216
        setne   sil
        and     sil, cl
        and     sil, dl
        mov     ecx, 4278190080
        or      rcx, 16777215
        cmp     rax, rcx
        seta    al
        and     al, sil
        ret
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Usually when several functions have the same assembly they are merged together. This not happening might indicate that the compiler did not understand that all of them do the same thing.&lt;&#x2F;p&gt;
&lt;p&gt;The assembly is an unrolled version of the iterator or loop. Note that the integer constants mask out some bits from a larger pattern like 0xFF00... There is a comparison for every bool. This feels suboptimal because all booleans being true has a single fixed byte pattern that we could compare against together. I try to get the compiler to understand this:&lt;&#x2F;p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;rust&quot;&gt;pub fn comparison(a: [bool; 5]) -&amp;gt; bool {
    a == [true; 5]
}

pub fn and(a: [bool; 5]) -&amp;gt; bool {
    a[0] &amp;amp; a[1] &amp;amp; a[2] &amp;amp; a[3] &amp;amp; a[4]
}
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;pre&gt;&lt;code data-lang=&quot;asm&quot;&gt;example::comparison:
        movabs  rax, 1099511627775
        and     rax, rdi
        movabs  rcx, 4311810305
        cmp     rax, rcx
        sete    al
        ret

example::and:
        not     rdi
        movabs  rax, 4311810305
        test    rdi, rax
        sete    al
        ret
&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;This is better. I&#x27;m not sure if &lt;code&gt;and&lt;&#x2F;code&gt; is optimal but it is the best version so far.&lt;&#x2F;p&gt;
&lt;h2 id=&quot;caveats&quot;&gt;caveats&lt;&#x2F;h2&gt;
&lt;p&gt;When I say that some code doesn&#x27;t optimize well or that some assembly is better, I mean that the code could be compiled into assembly that does the same thing in less time. If you are familiar with assembly, this can be intuited by looking at it. However, the quality of the assembly is not just a product of the instructions. It depends on other things like what CPU you have and what else is going on in the program. You need realistic benchmarks to determine whether some code is faster with high confidence. You might also care less about speed and more about the size of the resulting binary. These nuances do not matter for the examples in this post.&lt;&#x2F;p&gt;
&lt;p&gt;While I was able to rewrite code to improve generated assembly, none of the improvements are guaranteed. With the next compiler version both versions of the code might compile to the same assembly. Or the better version today might become the worse version tomorrow. This is an argument in favor of not worrying too much about the generated assembly and more about other metrics like code clarity. Still, for especially hot loops or especially bad assembly, making these adjustments can be worth it.&lt;&#x2F;p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>null terminated strings are incorrect</title>
        <published>2022-11-25T00:00:00+00:00</published>
        <updated>2026-05-10T00:00:00+00:00</updated>
        
        <author>
          <name>
            
              Unknown
            
          </name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://kttnr.net/blog/null-terminated-strings-are-incorrect/"/>
        <id>https://kttnr.net/blog/null-terminated-strings-are-incorrect/</id>
        
        <content type="html" xml:base="https://kttnr.net/blog/null-terminated-strings-are-incorrect/">&lt;h2 id=&quot;introduction&quot;&gt;introduction&lt;&#x2F;h2&gt;
&lt;p&gt;In the C programming language it is common to store text as null terminated strings. A null terminated string is a sequence of bytes ending in a null byte (&lt;code&gt;0x00&lt;&#x2F;code&gt;, more than one byte for wide characters) that represents a sequence of characters in some text encoding. Each character is mapped to one or more bytes according to the encoding.&lt;&#x2F;p&gt;
&lt;p&gt;C defines the following string types:&lt;&#x2F;p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;en.cppreference.com&#x2F;w&#x2F;c&#x2F;string&#x2F;byte&quot;&gt;byte&lt;&#x2F;a&gt;, example ASCII&lt;&#x2F;li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;en.cppreference.com&#x2F;w&#x2F;c&#x2F;string&#x2F;multibyte&quot;&gt;multibyte&lt;&#x2F;a&gt;, example UTF-8&lt;&#x2F;li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;en.cppreference.com&#x2F;w&#x2F;c&#x2F;string&#x2F;wide&quot;&gt;wide&lt;&#x2F;a&gt;, example UTF-32&lt;&#x2F;li&gt;
&lt;&#x2F;ul&gt;
&lt;p&gt;All of them are null terminated. Each comes with syntax for creating &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;en.cppreference.com&#x2F;w&#x2F;c&#x2F;language&#x2F;string_literal&quot;&gt;literals&lt;&#x2F;a&gt; in source code, and functions that operate on them like &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;en.cppreference.com&#x2F;w&#x2F;c&#x2F;string&#x2F;byte&#x2F;strcpy&quot;&gt;&lt;code&gt;strcpy&lt;&#x2F;code&gt;&lt;&#x2F;a&gt;.&lt;&#x2F;p&gt;
&lt;p&gt;For example, the literal &lt;code&gt;&quot;hello&quot;&lt;&#x2F;code&gt; is a null terminated byte string represented in memory as &lt;code&gt;0x68 0x65 0x6c 0x6c 0x6f 0x00&lt;&#x2F;code&gt;. The first 5 bytes come from the &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;en.cppreference.com&#x2F;w&#x2F;c&#x2F;language&#x2F;ascii&quot;&gt;ASCII&lt;&#x2F;a&gt; encoding. The next and last byte is the null byte. It doesn&#x27;t correspond to any character in the original text. It only indicates the end of the string.&lt;&#x2F;p&gt;
&lt;h2 id=&quot;problem&quot;&gt;problem&lt;&#x2F;h2&gt;
&lt;p&gt;Intuitively a programming language is expected to handle any string in the active encoding. It would be surprising if &lt;code&gt;hello world&lt;&#x2F;code&gt; was an acceptable string while &lt;code&gt;hello earth&lt;&#x2F;code&gt; was not.&lt;&#x2F;p&gt;
&lt;p&gt;C violates this expectation. It does so because it gives special meaning to the null byte. Encodings like ASCII and UTF-8 already assign meaning to the null byte. It encodes the &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Null_character&quot;&gt;null character&lt;&#x2F;a&gt;. What exactly the null character means is not relevant. It only matters that it is a valid character and encoded as &lt;code&gt;0x00&lt;&#x2F;code&gt;. This meaning clashes with the meaning C gives the null byte. A string supposed to contain the null character is instead cut short at its position.&lt;&#x2F;p&gt;
&lt;p&gt;For example, &lt;code&gt;hello\0world&lt;&#x2F;code&gt; (&lt;code&gt;\0&lt;&#x2F;code&gt; is the null character) is a valid ASCII and UTF-8 string that could be a literal in source code or read from a file. It can even be typed on the right kind of keyboard, just like pressing the enter key types the newline &lt;code&gt;\n&lt;&#x2F;code&gt; character. But the string functions in the C standard library cannot handle the null character correctly because they treat it as the end of the string like in the following examples:&lt;&#x2F;p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;strcpy&lt;&#x2F;code&gt; only copies &lt;code&gt;hello&lt;&#x2F;code&gt;.&lt;&#x2F;li&gt;
&lt;li&gt;&lt;code&gt;strcmp&lt;&#x2F;code&gt; returns that &lt;code&gt;hello\0world&lt;&#x2F;code&gt; and &lt;code&gt;hello\0earth&lt;&#x2F;code&gt; are equal.&lt;&#x2F;li&gt;
&lt;li&gt;&lt;code&gt;printf&lt;&#x2F;code&gt; only prints &lt;code&gt;hello&lt;&#x2F;code&gt;.&lt;&#x2F;li&gt;
&lt;&#x2F;ul&gt;
&lt;p&gt;This is not a bug in these functions. It is a fault in the C standard for designing the string types with null termination.&lt;&#x2F;p&gt;
&lt;h2 id=&quot;solution&quot;&gt;solution&lt;&#x2F;h2&gt;
&lt;p&gt;An alternative way to represent strings is with a pointer and a size. This is done in C++&#x27;s &lt;code&gt;std::string&lt;&#x2F;code&gt;, &lt;code&gt;std::string_view&lt;&#x2F;code&gt; and other languages. This representation does not have C&#x27;s problem and has other technical benefits unrelated to correctness that this post does not go into.&lt;&#x2F;p&gt;
&lt;p&gt;A historical reason that C chose the null byte representation is that it saves memory. Only one extra byte is needed. This is no longer a good reason and maybe was not one even then. The gain in efficiency is not worth the loss in correctness.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;Use better string types. Do not use C&#x27;s null terminated string functions. Use libraries that handle strings correctly.&lt;&#x2F;strong&gt;&lt;&#x2F;p&gt;
&lt;p&gt;I would like to link a C library that replicates the standard library&#x27;s string functions with pointer and size but I do not know one.&lt;&#x2F;p&gt;
&lt;p&gt;You can store string literals containing null bytes in arrays with &lt;code&gt;const char text[] = &quot;hello\0world&quot;;&lt;&#x2F;code&gt;. This gives you access to the size of the string where a pointer would not. The ending null byte is still added.&lt;&#x2F;p&gt;
&lt;h2 id=&quot;examples-of-problematic-software&quot;&gt;examples of problematic software&lt;&#x2F;h2&gt;
&lt;h3 id=&quot;sqlite&quot;&gt;SQLite&lt;&#x2F;h3&gt;
&lt;p&gt;SQLite &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;www.sqlite.org&#x2F;datatype3.html&quot;&gt;states&lt;&#x2F;a&gt; its string type stores UTF-8. This is incorrect because SQLite uses null terminated strings. See &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;sqlite.org&#x2F;c3ref&#x2F;bind_blob.html&quot;&gt;documentation&lt;&#x2F;a&gt; stating strings containing null bytes are undefined behavior. In practice they don&#x27;t cause errors but are sometimes silently cut off.&lt;&#x2F;p&gt;
&lt;details&gt;
&lt;summary&gt;documentation quote&lt;&#x2F;summary&gt;
&lt;blockquote&gt;
&lt;p&gt;In those routines that have a fourth argument, its value is the number of bytes in the parameter. To be clear: the value is the number of bytes in the value, not the number of characters. If the fourth parameter to sqlite3_bind_text() or sqlite3_bind_text16() is negative, then the length of the string is the number of bytes up to the first zero terminator. If the fourth parameter to sqlite3_bind_blob() is negative, then the behavior is undefined. If a non-negative fourth parameter is provided to sqlite3_bind_text() or sqlite3_bind_text16() or sqlite3_bind_text64() then that parameter must be the byte offset where the NUL terminator would occur assuming the string were NUL terminated. If any NUL characters occur at byte offsets less than the value of the fourth parameter then the resulting string value will contain embedded NULs. The result of expressions involving strings with embedded NULs is undefined.&lt;&#x2F;p&gt;
&lt;&#x2F;blockquote&gt;
&lt;&#x2F;details&gt;
&lt;details&gt;
&lt;summary&gt;program demonstrating null byte behavior&lt;&#x2F;summary&gt;
&lt;pre&gt;&lt;code data-lang=&quot;c&quot;&gt;&#x2F;&#x2F; Compile with `-std=C23`.

#include &amp;lt;assert.h&amp;gt;
#include &amp;lt;stdio.h&amp;gt;

#include &amp;quot;sqlite3.h&amp;quot;

int main() {
    sqlite3* db = nullptr;
    int result = 0;
    sqlite3_stmt* statement;

    const char* data = &amp;quot;first\0second&amp;quot;;
    const int data_len = 13;

    result = sqlite3_open_v2(
        &amp;quot;a&amp;quot;,
        &amp;amp;db,
        SQLITE_OPEN_CREATE | SQLITE_OPEN_READWRITE | SQLITE_OPEN_MEMORY,
        nullptr
    );
    assert(result == SQLITE_OK);

    result = sqlite3_exec(
        db,
        &amp;quot;CREATE TABLE table_(text TEXT, blob BLOB);&amp;quot;,
        nullptr,
        nullptr,
        nullptr
    );
    assert(result == SQLITE_OK);

    result = sqlite3_prepare_v3(
        db,
        &amp;quot;INSERT INTO table_(text, blob) VALUES (?1, ?2);&amp;quot;,
        -1,
        0,
        &amp;amp;statement,
        nullptr
    );
    assert(result == SQLITE_OK);
    result = sqlite3_bind_text(statement, 1, data, data_len, SQLITE_STATIC);
    assert(result == SQLITE_OK);
    result = sqlite3_bind_blob(statement, 2, data, data_len, SQLITE_STATIC);
    assert(result == SQLITE_OK);
    result = sqlite3_step(statement);
    assert(result == SQLITE_DONE);
    result = sqlite3_finalize(statement);
    assert(result == SQLITE_OK);

    result = sqlite3_prepare_v3(
        db,
        &amp;quot;SELECT text, blob FROM table_;&amp;quot;,
        -1,
        0,
        &amp;amp;statement,
        nullptr
    );
    assert(result == SQLITE_OK);
    result = sqlite3_step(statement);
    assert(result == SQLITE_ROW);
    const int text_size = sqlite3_column_bytes(statement, 0);
    const int blob_size = sqlite3_column_bytes(statement, 1);
    result = sqlite3_finalize(statement);
    assert(result == SQLITE_OK);

    result = sqlite3_prepare_v3(
        db,
        &amp;quot;SELECT CAST(text as BLOB), CAST(blob as TEXT) FROM table_;&amp;quot;,
        -1,
        0,
        &amp;amp;statement,
        nullptr
    );
    assert(result == SQLITE_OK);
    result = sqlite3_step(statement);
    assert(result == SQLITE_ROW);
    const int text_as_blob_size = sqlite3_column_bytes(statement, 0);
    const int blob_as_text_size = sqlite3_column_bytes(statement, 1);
    result = sqlite3_finalize(statement);
    assert(result == SQLITE_OK);

    result = sqlite3_prepare_v3(
        db,
        &amp;quot;SELECT text FROM table_ WHERE text LIKE &amp;#39;%second%&amp;#39;;&amp;quot;,
        -1,
        0,
        &amp;amp;statement,
        nullptr
    );
    assert(result == SQLITE_OK);
    result = sqlite3_step(statement);
    assert(result == SQLITE_ROW | result == SQLITE_DONE);
    const bool found_string = result == SQLITE_ROW;
    result = sqlite3_finalize(statement);
    assert(result == SQLITE_OK);

    result = sqlite3_close_v2(db);
    assert(result == SQLITE_OK);

    printf(&amp;quot;text size: %i\n&amp;quot;, text_size);
    printf(&amp;quot;blob size: %i\n&amp;quot;, blob_size);
    printf(&amp;quot;text as blob size: %i\n&amp;quot;, text_as_blob_size);
    printf(&amp;quot;blob as text size: %i\n&amp;quot;, blob_as_text_size);
    printf(&amp;quot;found string: %i\n&amp;quot;, found_string);
}

&#x2F;*
possible output:

text size: 13
blob size: 13
text as blob size: 13
blob as text size: 13
found string: 0
*&#x2F;


&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;&#x2F;details&gt;
&lt;h3 id=&quot;postgres&quot;&gt;Postgres&lt;&#x2F;h3&gt;
&lt;p&gt;PostgreSQL &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;www.postgresql.org&#x2F;docs&#x2F;15&#x2F;multibyte.html&quot;&gt;states&lt;&#x2F;a&gt; its string type can store UTF-8. This is incorrect because Postgres uses null terminated strings. This is handled better than in SQLite because Postgres &lt;a rel=&quot;external&quot; href=&quot;https:&#x2F;&#x2F;www.postgresql.org&#x2F;docs&#x2F;15&#x2F;datatype-character.html&quot;&gt;documents&lt;&#x2F;a&gt; (search for &lt;code&gt;NUL&lt;&#x2F;code&gt;) the restriction and errors when a string containing a null byte would be used.&lt;&#x2F;p&gt;
</content>
        
    </entry>
</feed>
