The important part is to learn the limits of any tool. Nowadays I no longer use jq for any long or complicated tasking. Filter and view data? jq is fine. Anything more and I just cook up a python script.
How do you get complex data structures to work? I was alienated from scripting on zsh because I wanted something like a dict and realised I would have to write my own implementation. Is there a work around for that?
This will do nicely - I had several workflows where I'd hit an API and get a massive super nested JSON as output; I'd use jq to get the specific data from the whole thing and do a bunch of stuff on this filtered data. I pretty much resigned to using python because I'd have successively complicated requirements and looking up how to do each new thing was slowing me down massively.
I recently noticed many people on lemmy have that thing rn. Why are they using it/is that autocorrecty thibgy or something? I didn't downvote them but i hate seeing this. And it's not just this letter
The english alphabet needs to be completely redone. We should bring back thorn, eth, and wynn. We should also increase the vowels to actually represent the crazy amount of vowel sounds we have, dipthongs are dumb. 5 vowels is not sufficient for 15+ phonemes.
I used to use it for math notation, so I'd welcome it's use again if I can keep using it as a placeholder for "then this happens" in between steps of functions.
my favorite awk snippet is !x[$0]++ which is like uniq but doesn't care about order. basically, it's equivalent to print_this_line = line_cache[$current_line] == 0; line_cache[$current_line] += 1; if $print_this_line then print $current_line end.
Looking at the above awk snippet, it'll retain order, though. So, sort will normally change the order. The awk snippet won't, just skip occurrences of a given line after the first. Depending upon the use case, that order retention could be pretty desireable.
I used awk for the first time today to find all the MD5 sums that matched an old file I had to get rid of. Still have no idea what awk was needed for. 😅 All my programming skill is in Python. Linux syntax is a weak point of mine.
To be fair, a lot of the programs don't use a single character, have multiple spaces between fields, and cut doesn't collapse whitespace characters, so you probably want something more like tr -s " "|cut -d" " -f3 if you want behavior like awk's field-splitting.
This is awesome! Looks like an LPI1 textbook. Never got the certification but I've seen a couple books about it and remember seeing examples like this one.
People are hating on Powershell way too much. I don't like its syntax really but it has a messy better approach to handling data in the terminal. We have nu and elvish nowadays but MS was really early with the concept and I think they learned from the shortcomings of POSIX compatible shells.
I really can't stress enough how much power and flexibility comes with an object oriented shell, especially with the dotnet type system behind it.
I think most people who hate it just do so either because it came from Microsoft (which... Yeah, that's understandable), or because it's a different way of thinking about it (and/or they spent a lot of effort learning how to parse data from strings effectively and hate that it's made easier?). But love or hate it, it is effective and powerful, and I find myself missing that when working with bash.
I use gawk all the fucking time, if you spend a lot of time in a terminal or parse text often it is definitely worth the investment. It is a fantastic tool for both one liners and full scripts. The gawk manual is short enough to digest in a day or two.
I remember when I first stumbled across this manual I was trying to look up a quick awk command and wound up reading the whole thing. It's really one of the better GNU manuals.
It's a Linux command-line program (awk). It's pre-installed practically everywhere, it's very powerful for string processing, but it also uses a fairly complex syntax.
As a result, not many people know how to really make use of it, but awk '{print $1}' is something you encounter fairly quickly when you need to get the first word in each line.
Hmm, but you have to install and run the Python environment for that. AWK is typically present on *NIX systems already. Python seem like overkill for basic text processing tasks.
I think the most I've done with awk is write a battery monitor applet with it, it involved parsing data from /sys and making choices based on it so I decided it was a decent choice
74 Comments
Lauchmelder@feddit.org · 66 pts · 1y
Why spend 30 seconds manually editing some text when you can spend 30 minutes clobbering together a pipeline involving awk, sed and jq
CheesyFox@lemmy.sdf.org · 21 pts · 1y
to be fair, out of those three, jq invokes the least existential dread in me
Tangent5280@lemmy.world · 4 pts · 1y
The important part is to learn the limits of any tool. Nowadays I no longer use jq for any long or complicated tasking. Filter and view data? jq is fine. Anything more and I just cook up a python script.
CheesyFox@lemmy.sdf.org · 5 pts · 1y
i'm more of a bash fan tbh. Ever since i started using linux, python started to irritate me
Tangent5280@lemmy.world · 4 pts · 1y
How do you get complex data structures to work? I was alienated from scripting on zsh because I wanted something like a dict and realised I would have to write my own implementation. Is there a work around for that?
tal@lemmy.today · 6 pts · 1y
I mean, there's a point in data structure complexity where it's useful to use Python.
But as to dicts, sure. You're looking for zsh's "associative array". Bash has it too.
zsh
bash
Tangent5280@lemmy.world · 4 pts · 1y
This will do nicely - I had several workflows where I'd hit an API and get a massive super nested JSON as output; I'd use jq to get the specific data from the whole thing and do a bunch of stuff on this filtered data. I pretty much resigned to using python because I'd have successively complicated requirements and looking up how to do each new thing was slowing me down massively.
GreenKnight23@lemmy.world · 8 pts · 1y
TIL I am an OP wizard.
grrgyle@slrpnk.net · 5 pts · 1y
Or 60 minutes making it all work just with jq functions.
freijon@lemmings.world · 0 pts · 364d
Sxan@piefed.zip · 44 pts · 1y
Ok, þe quote misplacement is really confusing. It's
How can you be so close to right about þis and still be wrong?
MalikMuaddibSoong@startrek.website · 28 pts · 1y
Honest answer: I’m sloppy on mobile
Better answer:
hddsx@lemmy.ca · 13 pts · 1y
Who downvoted this? If you use awk, you know Sxan is using the correct syntax.
teft@piefed.social · 31 pts · 1y
People have been downvoting him because he uses the letter thorn in his comments.
Some people will hate on anyone different.
mexicancartel@lemmy.dbzer0.com · 16 pts · 1y
I recently noticed many people on lemmy have that thing rn. Why are they using it/is that autocorrecty thibgy or something? I didn't downvote them but i hate seeing this. And it's not just this letter
Badabinski@kbin.earth · 15 pts · 1y
I'm not using it because it would be extremely inconvenient for me, but I think that the English language deserves to have the thorn returned to it.
teft@piefed.social · 10 pts · 1y
The english alphabet needs to be completely redone. We should bring back thorn, eth, and wynn. We should also increase the vowels to actually represent the crazy amount of vowel sounds we have, dipthongs are dumb. 5 vowels is not sufficient for 15+ phonemes.
derfunkatron@lemmy.world · 3 pts · 1y
Let’s capitalize nouns again while we’re at it.
possiblylinux127@lemmy.zip · 5 pts · 1y
Please no
I have a hard enough time with English already
scathliath@lemmy.dbzer0.com · 3 pts · 1y
I used to use it for math notation, so I'd welcome it's use again if I can keep using it as a placeholder for "then this happens" in between steps of functions.
somerandomperson@lemmy.dbzer0.com · 8 pts · 1y
It's to confuse scrapers.
Gutek8134@lemmy.world · 12 pts · 1y
It's going to be fun for etymologists 100 years from now
FiskFisk33@startrek.website · 3 pts · 1y
I see what you did þere
Sxan@piefed.zip · 1 pts · 1y
Ssshhh, maybe no-one will notice!
lime@feddit.nu · 22 pts · 1y
my favorite awk snippet is
!x[$0]++which is likeuniqbut doesn't care about order. basically, it's equivalent toprint_this_line = line_cache[$current_line] == 0; line_cache[$current_line] += 1; if $print_this_line then print $current_line end.really useful for those long spammy logs.
grrgyle@slrpnk.net · 3 pts · 1y
Oh that's very interesting. I usually do
sort --uniqueorsort [...] | uniqif I need specific sorting logic (like by size on disk, etc).tal@lemmy.today · 6 pts · 1y
Looking at the above awk snippet, it'll retain order, though. So, sort will normally change the order. The awk snippet won't, just skip occurrences of a given line after the first. Depending upon the use case, that order retention could be pretty desireable.
OmnislashIsACloudApp@lemmy.world · 3 pts · 1y
same, that statement saved me so much effort
tdawg@lemmy.world · 19 pts · 1y
In all my years I've only used more than that a handful of times. Just don't need it really
Now jq on the other hand...
harsh3466@lemmy.ml · 7 pts · 1y
jqis indispensableCubitOom@infosec.pub · 18 pts · 1y
I've become a person that uses awk instead of grep, sed, cut, head, tail, cat, perl, or bashisms
OpenStars@piefed.social · 14 pts · 1y
The stage of your degeneracy will involve learning PERL.
Edit: one-liners FTW! 😁🐪
SkaveRat@discuss.tchncs.de · 7 pts · 1y
I could try to learn awk while also trying to debug the annoying problem I'm trying to solve, orrr....
cutandgrepit isMalikMuaddibSoong@startrek.website · 6 pts · 1y
sort | uniq -c has entered the chat 🤣
poinck@lemmy.world · 2 pts · 1y
otacon239@lemmy.world · 15 pts · 1y
I used awk for the first time today to find all the MD5 sums that matched an old file I had to get rid of. Still have no idea what awk was needed for. 😅 All my programming skill is in Python. Linux syntax is a weak point of mine.
Ephera@lemmy.ml · 6 pts · 1y
Probably the very same thing that the post talks about, which is extracting the first word of a line of text.
The output of
md5sumlooks like this:So, it lists the checksum and then the file name, but you wanted just the checksum.
UltraBlack@lemmy.world · 14 pts · 1y
All my homies use dubious regex
TheGingerNut@lemmy.blahaj.zone · 13 pts · 1y
Honestly I think 90% of people would never use awk if there was a simple preinstalled command for "print the nth column"
OpenStars@piefed.social · 16 pts · 1y
cut?
tal@lemmy.today · 16 pts · 1y
To be fair, a lot of the programs don't use a single character, have multiple spaces between fields, and
cutdoesn't collapse whitespace characters, so you probably want something more liketr -s " "|cut -d" " -f3if you want behavior likeawk's field-splitting.TechLich@lemmy.world · 8 pts · 1y
I never understood why so many bash scripts pipe grep to awk when regex is one of its main strengths.
Like... Why
grep ^nvme0n1 | awk '{print $3}'over just
awk '/^nvme0n1/ {print $3}'FooBarrington@lemmy.world · 12 pts · 1y
Because by the time I use
awkagain, I've completely forgotten that it supports this stuff, and the discoverability is horrendous.Though I'd happily fix it if ShellCheck warned against this...
ThunderLegend@sh.itjust.works · 2 pts · 1y
This is awesome! Looks like an LPI1 textbook. Never got the certification but I've seen a couple books about it and remember seeing examples like this one.
MalikMuaddibSoong@startrek.website · 1 pts · 1y
cut and tr are like the wonder twins of text munging
laurelraven@lemmy.zip · 4 pts · 1y
This is definitely somewhere that PowerShell shines, all of that is built in and really easy to use
Laser@feddit.org · 1 pts · 1y
People are hating on Powershell way too much. I don't like its syntax really but it has a messy better approach to handling data in the terminal. We have nu and elvish nowadays but MS was really early with the concept and I think they learned from the shortcomings of POSIX compatible shells.
laurelraven@lemmy.zip · 2 pts · 1y
I really can't stress enough how much power and flexibility comes with an object oriented shell, especially with the dotnet type system behind it.
I think most people who hate it just do so either because it came from Microsoft (which... Yeah, that's understandable), or because it's a different way of thinking about it (and/or they spent a lot of effort learning how to parse data from strings effectively and hate that it's made easier?). But love or hate it, it is effective and powerful, and I find myself missing that when working with bash.
cmgvd3lw@discuss.tchncs.de · 11 pts · 1y
You can even do sum with awk, you don't need excel
tal@lemmy.today · 3 pts · 1y
num-utilshasnumsum.h4x0r@lemmy.dbzer0.com · 11 pts · 1y
I use
gawkall the fucking time, if you spend a lot of time in a terminal or parse text often it is definitely worth the investment. It is a fantastic tool for both one liners and full scripts. Thegawkmanual is short enough to digest in a day or two.https://www.gnu.org/software/gawk/manual/gawk.html
bulwark@lemmy.world · 6 pts · 1y
I remember when I first stumbled across this manual I was trying to look up a quick awk command and wound up reading the whole thing. It's really one of the better GNU manuals.
expr@programming.dev · 9 pts · 1y
cut -d ' ' -f1master racesomeguy3@lemmy.world · 9 pts · 1y
What's an awk?
TheBat@lemmy.world · 27 pts · 1y
Ephera@lemmy.ml · 21 pts · 1y
It's a Linux command-line program (
awk). It's pre-installed practically everywhere, it's very powerful for string processing, but it also uses a fairly complex syntax.As a result, not many people know how to really make use of it, but
awk '{print $1}'is something you encounter fairly quickly when you need to get the first word in each line.tofu@lemmy.nocturnal.garden · 14 pts · 1y
awkward
ratel@mander.xyz · 9 pts · 1y
That's where they send you when the syntax drives you insane.
tal@lemmy.today · 8 pts · 1y
https://en.wikipedia.org/wiki/AWK
SendMePhotos@lemmy.world · 6 pts · 1y
awk yeah
driving_crooner@lemmy.eco.br · -12 pts · 1y
Amazon Web Keynotes. It's a programming language for server administration.
juliebean@lemmy.zip · 8 pts · 1y
uh, no.
Gyroplast@pawb.social · 4 pts · 1y
ITT: whooosh
possiblylinux127@lemmy.zip · 3 pts · 1y
Wut
neuracnu@lemmy.blahaj.zone · 9 pts · 1y
10 PRINT BUTTS
20 GOTO 10
pelya@lemmy.world · 8 pts · 1y
Everything you do with
awk, you can do withpython, and it will also be readable.SkaveRat@discuss.tchncs.de · 9 pts · 1y
and
perl, if you want it less readablepossiblylinux127@lemmy.zip · 3 pts · 1y
Or you are old and crazy
laurelraven@lemmy.zip · 2 pts · 1y
Or PowerShell if you want it extra verbose
NaibofTabr@infosec.pub · 7 pts · 1y
Hmm, but you have to install and run the Python environment for that. AWK is typically present on *NIX systems already. Python seem like overkill for basic text processing tasks.
pelya@lemmy.world · 3 pts · 1y
On Debian the
pythonis preinstalled.GreenKnight23@lemmy.world · 8 pts · 1y
joke so dark I had to turn up my screen brightness to enjoy it.
grrgyle@slrpnk.net · 6 pts · 1y
Hey I throw a
/^regexp.*/ {print $NF}in there sometimes!...but yes, it's mostly
print $1—but only because I mix up the parameters whenever I try to usecut!Doomsider@lemmy.world · 6 pts · 1y
My five thousand line bash script can do things that one hundred thousand lines of code could not do.
On the brightside, at least script monkeys can now look down on vibe coders.
mumblerfish@lemmy.world · 4 pts · 1y
-- [x] I'm in this picture and don't like it.
KSPAtlas@sopuli.xyz · 2 pts · 1y
I think the most I've done with awk is write a battery monitor applet with it, it involved parsing data from /sys and making choices based on it so I decided it was a decent choice
grandma@sh.itjust.works · 1 pts · 1y
Nushell enjoyers represent