Hacker Public Radio

HPR4201: Today I learnt (2024-08-23)


Listen Later

TIL 1:
Incrementing times with the date command
  • I have been working on an HPR project which is restoring external
  • files to shows which lost them when we migrated to the current static
    site.
  • As I make changes I want to be able to check that they are correct,
  • but to make this check I need to wait for the next update of the static
    site.
  • Ken was away recently, and set up a cron job to refresh
  • the site every three hours. Each show page shows the refresh time in UTC
    form in the header.
  • Being a bit numerically challenged I wanted a way of computing the
  • next refresh time in my timezone from the previous refresh time.
  • The GNU date command accepts a date and time expression
  • after the -d option (or using the
    alternative--date=STRING option). The contents of the
    STRING here are very flexible but quite complex since you
    can include time zone data, offsets, day and month names, etc. See the
    links below for links to the GNU manual.
  • My first attempt used the date command like this and
  • got the wrong answer (using the output format +%T which
    writes the time in a default form):
    $ date -d '16:27:16 + 3 hours' +%T
    15:27:16
    • It is not clear why this fails, but the GNU function which parses
    • these date parameters is obviously confused. The second try included the
      time zone after the time, and worked better, but is a little
      confusing:
      $ date -d '16:27:16 UTC + 3 hours' +%T
      20:27:16
      • The time returned is local time for me. The date
      • command has added three hours to the UTC date to get 19:27:16, but since
        I am in the UK, which is in DST (called BST - British Summer Time - UTC
        plus 1 hour), an hour is added.
      • The final try used the -u option which writes UTC
      • time:
        $ date -u -d '16:27:16 UTC + 3 hours' +%T
        19:27:16
        • I actually ended up using and re-using these commands (though a
        • script would have been better):
          $ current='06:27:55'
          $ next=$(date -u -d "${current}UTC + 3 hours 3 minutes" +%T); echo "$next UTC / $(date -d "${next} UTC" +'%T %Z')"
          09:30:55 UTC / 10:30:55 BST
          $ current=$next
          $ next=$(date -u -d "${current}UTC + 3 hours 3 minutes" +%T); echo "$next UTC / $(date -d "${next} UTC" +'%T %Z')"
          12:33:55 UTC / 13:33:55 BST
          $ current=$next
          $ next=$(date -u -d "${current}UTC + 3 hours 3 minutes" +%T); echo "$next UTC / $(date -d "${next} UTC" +'%T %Z')"
          15:36:55 UTC / 16:36:55 BST
          TIL 1: Links
          • GNU
          • Coreutils manual
            • System
            • context
              • date:
              • Print or set system date and time
                • Date
                • input formats (The quotation at the start of this section of the
                  manual is quite interesting!)
                • Date
                • examples
                  TIL 2: Merging lines of
                  files with paste
                  • While processing and "repairing" shows I came across the need to
                  • generate a list of show numbers separated by commas. In the past I have
                    loaded these into a Bash array and turned them into a comma-delimited
                    string, using the parameter substitution capabilities of Bash which can
                    add a comma to each element. The trouble with this is that it leaves a
                    trailing comma which has to be removed. I stumbled upon
                    paste as an alternative way of doing this.
                  • The GNU paste command is another from the GNU
                  • Coreutils group. This one merges lines of files. Its synopsis
                    is:
                    paste [OPTION]... [FILE]...
                    • It merges lines consisting of the corresponding lines from each file
                    • provided as an argument, by default separated by TABs, and writes them
                      to standard output. This means it produces lines consisting of the first
                      line from each of the files, separated by tabs, then the second lines,
                      and so on.
                    • Any of the files can be linked to standard in by using a
                    • - (hyphen) as the file name.
                    • The delimiters can be changed with the -d LIST or
                    • --delimiters=LIST option. The use of a list of delimiters
                      causes the characters in the list to be used sequentially for each
                      delimiter.
                    • The merged lines can be visualised as rows in a matrix, where each
                    • file provides a column.
                    • The "matrix" is rotated by using the -s or
                    • --serial option. Here the lines from one file at a time are
                      merged - the files are processed serially rather than in
                      parallel.
                    • The paste command can be used to generate the
                    • comma-delimited list I wanted by using the options -s and
                      -d ',':
                      $ printf '%s\n' {1..10} | paste -s -d, -
                      1,2,3,4,5,6,7,8,9,10
                      • Note that you can't get the same result with
                      • echo {1..10} because all the numbers will be written to one
                        line rather than being the separate lines that paste
                        requires.
                      • The file arguments to paste may also be Bash
                      • process substitution expressions:
                        $ paste -s -d, <(printf '%s\n' {1..10})
                        1,2,3,4,5,6,7,8,9,10
                        • This means you can generate more complex output by using multiple
                        • process substitution expressions where each is seen as a
                          file:
                          $ paste -d'|' <(printf '%d\n' {1..7}) <(printf '%s\n' {A..G}) <(printf '%d\n' {100..106})
                          1|A|100
                          2|B|101
                          3|C|102
                          4|D|103
                          5|E|104
                          6|F|105
                          7|G|106
                          • Note how using the -s option "rotates" this:
                          • $ paste -s -d'|' <(printf '%d\n' {1..7}) <(printf '%s\n' {A..G}) <(printf '%d\n' {100..106})
                            1|2|3|4|5|6|7
                            A|B|C|D|E|F|G
                            100|101|102|103|104|105|106
                            • This is what I was actually trying to do, so I could feed show
                            • numbers to another script which will accept a CSV list as an
                              option:
                              $ echo "select episode_id from repairs where repair_date is null order by episode_id desc limit 5" |\
                              sqlite3 -list ~/HPR/InternetArchive/ia.db | paste -s -d',' -
                              1959,1952,1951,1946,1941
                              TIL 2: Links
                              • GNU
                              • Coreutils manual
                                • Operating
                                • on fields
                                  • paste:
                                  • Merge lines of files
                                    ...more
                                    View all episodesView all episodes
                                    Download on the App Store

                                    Hacker Public RadioBy Hacker Public Radio

                                    • 4.2
                                    • 4.2
                                    • 4.2
                                    • 4.2
                                    • 4.2

                                    4.2

                                    34 ratings


                                    More shows like Hacker Public Radio

                                    View all
                                    The Changelog: Software Development, Open Source by Changelog Media

                                    The Changelog: Software Development, Open Source

                                    289 Listeners

                                    Defensive Security Podcast - Malware, Hacking, Cyber Security & Infosec by Jerry Bell and Andrew Kalat

                                    Defensive Security Podcast - Malware, Hacking, Cyber Security & Infosec

                                    373 Listeners

                                    LINUX Unplugged by Jupiter Broadcasting

                                    LINUX Unplugged

                                    268 Listeners

                                    SANS Internet Stormcenter Daily Cyber Security Podcast (Stormcast) by Johannes B. Ullrich

                                    SANS Internet Stormcenter Daily Cyber Security Podcast (Stormcast)

                                    653 Listeners

                                    Curious Cases by BBC Radio 4

                                    Curious Cases

                                    826 Listeners

                                    The Strong Towns Podcast by Strong Towns

                                    The Strong Towns Podcast

                                    423 Listeners

                                    Late Night Linux by The Late Night Linux Family

                                    Late Night Linux

                                    164 Listeners

                                    Darknet Diaries by Jack Rhysider

                                    Darknet Diaries

                                    8,039 Listeners

                                    Cybersecurity Today by Jim Love

                                    Cybersecurity Today

                                    181 Listeners

                                    CISO Series Podcast by David Spark, Mike Johnson, and Andy Ellis

                                    CISO Series Podcast

                                    189 Listeners

                                    TechCrunch Daily Crunch by TechCrunch

                                    TechCrunch Daily Crunch

                                    42 Listeners

                                    Strict Scrutiny by Crooked Media

                                    Strict Scrutiny

                                    5,809 Listeners

                                    2.5 Admins by The Late Night Linux Family

                                    2.5 Admins

                                    98 Listeners

                                    Cyber Security Headlines by CISO Series

                                    Cyber Security Headlines

                                    138 Listeners

                                    What the Hack? by DeleteMe

                                    What the Hack?

                                    221 Listeners