this post was submitted on 15 Jan 2025

76 points (97.5% liked)

Programming

17823 readers

396 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 2 years ago

MODERATORS

snowe@programming.dev

Ategon@programming.dev

MaungaHikoi@lemmy.nz

Am I crazy in thinking that bash is good enough for production? (lemm.ee)

submitted 1 day ago by Badland9085@lemm.ee to c/programming@programming.dev

104 comments fedilink hide all child comments

This may make some people pull their hair out, but I’d love to hear some arguments. I’ve had the impression that people really don’t like bash, not from here, but just from people I’ve worked with.

There was a task at work where we wanted something that’ll run on a regular basis, and doesn’t do anything complex aside from reading from the database and sending the output to some web API. Pretty common these days.

I can’t think of a simpler scripting language to use than bash. Here are my reasons:

Reading from the environment is easy, and so is falling back to some value; just do ${VAR:-fallback}; no need to write another if-statement to check for nullity. Wanna check if a variable’s set to something expected? if [[ <test goes here> ]]; then <handle>; fi
Reading from arguments is also straightforward; instead of a import os; os.args[1] in Python, you just do $1.
Sending a file via HTTP as part of an application/x-www-form-urlencoded request is super easy with curl. In most programming languages, you’d have to manually open the file, read them into bytes, before putting it into your request for the http library that you need to import. curl already does all that.
Need to read from a curl response and it’s JSON? Reach for jq.
Instead of having to set up a connection object/instance to your database, give sqlite, psql, duckdb or whichever cli db client a connection string with your query and be on your way.
Shipping is… fairly easy? Especially if docker is common in your infrastructure. Pull Ubuntu or debian or alpine, install your dependencies through the package manager, and you’re good to go. If you stay within Linux and don’t have to deal with differences in bash and core utilities between different OSes (looking at you macOS), and assuming you tried to not to do anything too crazy and bring in necessary dependencies in the form of calling them, it should be fairly portable.

Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.

For most bash gotchas, shellcheck does a great job at warning you about them, and telling how to address those gotchas.

There are probably a bunch of other considerations but I can’t think of them off the top of my head, but I’ve addressed a bunch before.

So what’s the dealeo? What am I missing that may not actually be addressable?

(page 2) 50 comments

sorted by: hot top controversial new old

[–] syklemil@discuss.tchncs.de 12 points 1 day ago (4 children)

At the level you're describing it's fine. Preferably use shellcheck and set -euo pipefail to make it more normal.

But once I have any of:

nested control structures, or
multiple functions, or
have to think about handling anything else than simple strings that other programs manipulate (including thinking about bash arrays or IFS), or
bash scoping,
producing my own formatted logs at different log levels,

I'm on to Python or something else. It's better to get off bash before you have to juggle complexity in it.

load more comments (4 replies)

[–] Ephera@lemmy.ml 6 points 1 day ago (1 children)

Wanna check if a variable’s set to something expected? if [[ <test goes here> ]]; then <handle>; fi

Hey, you can't just leave out "test goes here". That's worst part by a long shot.
The rest of the syntax, I will have to look up every time I try to write it, but at least I can mostly guess what it does when reading. The test syntax on the other hand is just impossible to read without looking it up.

I also don't actually know how to look that up for the double brackets, so that's fun. For the single bracket, it took me years to learn that that's actually a command and you can do man [ to view the documentation.

[–] Badland9085@lemm.ee 4 points 1 day ago (2 children)

To be fair, you don’t always have to use the [[ syntax. I know I don’t, e.g. if I’m just looking for a command that returns 1 or 0, which happens quite a bit if you get to use grep.

That said, man test is my friend.

But I’ve also gotten so used to using it that I remember -z and -n by heart :P

load more comments (2 replies)

[–] friend_of_satan@lemmy.world 3 points 1 day ago

Dude, pihole is bash.

https://github.com/pi-hole/pi-hole/blob/master/pihole

[–] MajorHavoc@programming.dev 11 points 1 day ago* (last edited 1 day ago) (2 children)

A few responses for you:

I deeply despise bash (edit: this was hyperbole. I also deeply appreciate bash, as is appropriate for something that has made my life better for free!). That Linux shell defaults settled on it is an embarrassment to the entire open source community. (Edit: but Lexers and Parsers are hard! You don't see me fixing it, so yes, I'll give it a break. I still have to be discerning for production use, of course.)
Yes, Bash is good enough for production. It is the world's current default shell. As long as we avoid it's fancier features (which all suck for production use), a quick bash script is often the most reasonable choice.
For the love of all that is holy, put your own personal phone number and no one else's in the script, if you choose to use bash to access a datatbase. There's thousands of routine ways that database access can hiccup, and bash is suitable to help you diagnose approximately 0% of them.
If I found out a colleague had used bash for database access in a context that I would be expected to co-maintain, I would start by plotting their demise, and then talk myself down to having a severe conversation with them - after I changed it immediately to something else, in production, ignoring all change protocols. (Invoking emergency change protocols.)

Edit: I can't even respond to the security concerns aspect of this. Choice of security tool affects the quality of protection. In this unfortunate analogy, Bash is "the pull out method". Don't do that anywhere that it matters, or anywhere that one can be fired for security violations.

(Edit 2: Others have mentioned invoking SQL DB cleanup scripts from bash. I have no problem with that. Letting bash or cron tell the DB and a static bit of SQL to do their usual thing has been fine for me, as well. The nightmare scenario I was imagining was bash gathering various inputs to the SQL and then invoking them. I've had that pattern blow up in my face, and had a devil of a time putting together what went wrong. It also comes with security concerns, as bash is normally a completely trusted running environment, and database input often come from untrusted sources.)

[–] NegativeLookBehind@lemmy.world 8 points 1 day ago (1 children)

Why internet man hate Bash? Bash do many thing. Make computer work.

[–] MajorHavoc@programming.dev 7 points 1 day ago* (last edited 1 day ago) (1 children)

I actually (also) love bash, and use it like crazy.

What I really hate is that bash is so locked in legacy that it's bad features (on a scripting language scale, which isn't fair) (and of which there are too many to enumerate) are now locked in permanently.

I also hate how convention has kept other shells from replacing bash's worst features with better modern alternatives.

To some extent, I'm railing against how hard it is to write a good Lexer and a Parser, honestly. Now that bash is stable, there's little interest in improving it. Particularly since one can just invoke a better scripting language for complex work.

I mourn the sweet spot that Perl occupies, that Bash and Python sit on either side of, looking longingly across the gap that separated their practical use cases.

I have lost hope that Python will achieve shell script levels of pragmatism. Although the invoke library is a frigging cool attempt.

But I hold on to my sorrow and anger that Bash hasn't bridged the gap, and never will, because whatever it can invoke, it's methods of responding to that invocation are trapped in messes like "if...fi".

[–] Badland9085@lemm.ee 2 points 1 day ago (1 children)

What do you suppose bash could do here? When a program reaches some critical mass in terms of adoption, all your bugs and features are features of your program, and, love it or hate it, somebody’s day is going to be ruined if you do your bug fixes, unless, of course, it’s a fix for something that clearly doesn’t work in the very sense of the word.

I’m sure there’s space for a clear alternative to arise though, as far as scripting languages go. Whether we’ll see that anytime soon is hard to tell, cause yeah, a good lexer and parser in the scripting landscape is hard work.

[–] MajorHavoc@programming.dev 4 points 1 day ago

What do you suppose bash could do here?

For the love of all that is holy, it's not 1970, we don't need to continue to tolerate "if ... fi"
Really everything about how bash handles logic bridging multiple lines of a file. (loops, error handling, etc)

I’m sure there’s space for a clear alternative to arise though, as far as scripting languages go.

The first great alternative/attempt does exist, in PowerShell. (Honorable mention to Zsh, but I find it has most of the same issues as bash without gaining the killer features of pwsh.)

But I'm a cranky old person so I despise (and deeply appreciate!) PowerShell for a completely different set of reasons.

At the moment I use whichever gets the job done, but I would love to stop switching quite so often.

I hold more hope that PowerShell will grow to bridge the gap than that a fork of bash will. The big thing PowerShell lacks is bash's extra decades of debugging and refinement.

[–] Badland9085@lemm.ee 2 points 1 day ago* (last edited 1 day ago) (1 children)

Could you explain those db connection hiccups you’ve seen?

[–] MajorHavoc@programming.dev 1 points 1 day ago (1 children)

Sure.

I'll pick on postgres because it's popular. But I have found that most databases have a similar number of error codes.

https://www.postgresql.org/docs/current/errcodes-appendix.html

It's not an specific error that's the issue, it's the sheer variety of ways things can go wrong, combined with bash not having been architected with the database access use case in mind.

[–] Badland9085@lemm.ee 1 points 1 day ago (2 children)

I find this argument somewhat weak. You are not going to run into the vast majority of those errors (in fact, some of them are not even errors, and you will probably never run into some of those errors as Postgres will not return them, eg some error codes from the sql standard). Many of them will only trigger if you do specific things: you started a transaction, you’ll have to handle the possible errors that comes with having a transaction.

There are lots of reasons to never use bash to connect to a db to do things. Here are a couple I think of that I think are fairly basic that some may think they can just do in bash.

Write to more than 1 table.
Write to a table that has triggers, knowing that you may get a trigger failure.
Use transactions.
Calling a stored procedure that will raise exceptions.
Accepting user input to write that into a table.

One case that I think is fine to use bash and connect to a db is when all you need to do a SELECT. You can test your statement in your db manager of choice, and bring that into bash. If you need input sanitization to filter results, stop, and use a language with a proper library. Otherwise, all the failure cases I can think of are: a) connection fails for whatever reason, in which case you don’t get your data, you get an exit code of 1, log to stderr, move on, b) your query failed cause of bad sql, in which case, well, go back to your dev loop, no?

This is why I asked what sort of problems have you ran into before, assuming you haven’t been doing risky things with the connection. I’m sorry, but I must say that I’m fairly disappointed by your reply.

[–] MajorHavoc@programming.dev 1 points 1 day ago* (last edited 1 day ago)

This is why I asked what sort of problems have you ran into before,

Lol. I'm fucking old. I don't remember details.

assuming you haven’t been doing risky things with the connection.

Ha! Not a safe assumption, though. I've maintained even more shitty code than I've written, and that's a lot! Lol.

[–] MajorHavoc@programming.dev 1 points 1 day ago* (last edited 1 day ago) (3 children)

I find this argument somewhat weak.

Lol. Me too. I was just trying to give the shorthand version.

Your explanation is much better.

Edit: but it doesn't sound like you really needed a detailed answer from me, anyway.

load more comments (3 replies)

[–] Kissaki@programming.dev 6 points 1 day ago (2 children)

In your own description you added a bunch of considerations, requirements of following specific practices, having specific knowledge, and a ton of environmental requirements.

For simple scripts or duck tape schedules all of that is fine. For anything else, I would be at least mindful if not skeptical of bash being a good tool for the job.

Bash is installed on all linux systems. I would not be very concerned about some dependencies like sqlite, if that is what you're using. But very concerned about others, like jq, which is an additional tool and requirement where you or others will eventually struggle with diffuse dependencies or managing a managed environment.

Even if you query sqlite or whatever tool with the command line query tool, you have to be aware that getting a value like that into bash means you lose a lot of typing and structure information. That's fine if you get only one or very few values. But I would have strong aversions when it goes beyond that.

You seem to be familiar with Bash syntax. But others may not be. It's not a simple syntax to get into and intuitively understand without mistakes. There's too many alternatives of if-ing and comparing values. It ends up as magic. In your example, if you read code, you may guess that :- means fallback, but it's not necessarily obvious. And certainly not other magic flags and operators.

As an anecdote, I guess the most complex thing I have done with Bash was scripting a deployment and starting test-runs onto a distributed system (and I think collecting results? I don't remember). Bash was available and copying and starting processes via ssh was simple and robust enough. Notably, the scope and env requirements were very limited.

[–] palordrolap@fedia.io 8 points 1 day ago (1 children)

You seem to be familiar with Bash syntax. But others may not be.

If by this you mean that the Bash syntax for doing certain things is horrible and that it could be expressed more clearly in something else, then yes, I agree, otherwise I'm not sure this is a problem on the same level as others.

OP could pick any language and have the same problem. Except maybe Python, but even that strays into symbolic line noise once a project gets big enough.

Either way, comments can be helpful when strange constructs are used. There are comments in my own Bash scripts that say what a line is doing rather than just why precisely because of this.

But I think the main issue with Bash (and maybe other shells), is that it's parsed and run line by line. There's nothing like a full script syntax check before the script is run, which most other languages provide as a bare minimum.

[–] Kissaki@programming.dev 1 points 23 hours ago

OP could pick any language and have the same problem. Except maybe Python, but even that strays into symbolic line noise once a project gets big enough.

Personally, I don't see python far off from bash. Decent for small scripts, bad for anything bigger. While not necessarily natively available, it's readily available and more portable (Windows), and has a rich library ecosystem.

Personally, I dislike the indent syntax. And the various tooling and complexities don't feel approachable or stable, and structuring not good.

But maybe that's me. Many people seem to enjoy or reach for python even for complex systems.

More structured and stable programming languages do not have these issues.

[–] Badland9085@lemm.ee 2 points 1 day ago (2 children)

As one other comment mentioned, unfamiliarity with a particular language isn’t restricted to just bash. I could say the same for someone who only dabbles in C being made to read through Python. What’s this @decorator thing? Or what’s f"Some string: {variable}" supposed to do, and how’s memory being allocated here? It’s a domain, and we aren’t expected to know every single domain out there.

And your mention of losing typing and structure information is… ehh… somewhat of a weird argument. There are many cases where you don’t care about the contents of an output and only care about the process of spitting out that output being a success or failure, and that’s bread and butter in shell scripts. Need to move some files, either locally or over a network, bash is good for most cases. If you do need something just a teeny bit more, like whether some key string or pattern exists in the output, there’s grep. Need to do basic string replacements? sed or awk. Of course, all that depends on how familiar you or your teammates are with each of those tools. If nearly half the team are not, stop using bash right there and write it in something else the team’s familiar with, no questions there.

This is somewhat of an aside, but jq is actually pretty well-known and rather heavily relied upon at this point. Not to the point of say sqlite, but definitely more than, say, grep alternatives like ripgrep. I’ve seen it used quite often in deployment scripts, especially when interfaced with some system that replies with a json output, which seems like an increasingly common data format that’s available in shell scripting.

load more comments (2 replies)

[–] GammaGames@beehaw.org 8 points 1 day ago* (last edited 1 day ago) (3 children)

I agree with your points, except if the script ever needs maintaining by someone else’s they will curse you and if it gets much more complicated it can quickly become spaghetti. But I do have a fair number of bash scripts running on cron jobs, sometimes its simplicity is unbeatable!

Personally though the language I reach for when I need a script is Python with the click library, it handles arguments and is really easy to work with. If you want to keep python deps down you can also use the sh module to run system commands like they’re regular python, pretty handy

load more comments (3 replies)

[–] SilverShark@programming.dev 8 points 1 day ago (1 children)

I don't disagree with this, and honestly I would probably support just using bash like you said if I was in a team where this was suggested.

I think no matter how simple a task is there are always a few things people will eventually want to do with it:

Reproduce it locally
Run unit tests, integration tests, smoke tests, whatever tests
Expand it to do more complex things or make it more dynamic
Monitor it in tools like Datadog

If you have a whole project already written in Python, Go, Rust, Java, etc, then just writing more code in this project might be simpler, because all the tooling and methodology is already integrated. A script might not be so present for many developers who focus more on the code base, and as such out of sight out of mind sets in, and no one even knows about the script.

There is also the consideration that many people simply dislike bash since it's an odd language and many feel it's difficult to do simple things with it.

due to these reasons, although I would agree with making the script, I would also be inclined to have the script temporarily while another solution is being implemented.

[–] Badland9085@lemm.ee 1 points 1 day ago

I don’t necessarily agree that all simple tasks will lead to the need for a test suite to accommodate more complex requirements. If it does reach that point,

Your simple bash script has and is already providing basic value.
You can (and should) move onto a more robust language to do more complicated things and bring in a test suite, all while you have something functional and delivering value.

I also don’t agree that you can just solder on whatever small task you have to whatever systems you already have up and running. That’s how you make a Frankenstein. Someone at some point will have to come do something about your little section because it started breaking, or causing other things to break. It could be throwing error messages because somebody changed the underlying db schema. It could be calling and retrying a network call and due to, perhaps, poorly configured backoff strategy, you’re tripping up monitoring alerts.

That said, I do agree on it suitable for temporary tasks.

[–] 31337@sh.itjust.works 7 points 1 day ago

It's ok for very small scripts that are easy to reason through. I've used it extensively in CI/CD, just because we were using Jenkins for that and it was the path of least resistance. I do not like the language though.

[–] thirteene@lemmy.world 2 points 1 day ago (1 children)

Pretty much all languages are middleware, and most of the original code was shell/bash. All new employees in platform/devops want to immediately push their preferred language, they want java and rust environments. It's a pretty safe bet if they insist on using a specific language; then they don't know how awk or sed. Bash has all the tools you need, but good developers understand you write libraries for functionality that's missing. Modern languages like Python have been widely adopted and has a friendlier onboarding and will save you time though.

[–] BatmanAoD@programming.dev 2 points 1 day ago (1 children)

Pretty much all languages are middleware, and most of the original code was shell/bash.

What? I genuinely do not know what you mean by this.

[–] thirteene@lemmy.world 0 points 16 hours ago (1 children)

2 parts:

All languages are middleware. Unless you write in assembly, whatever you write isn't directly being executed, they are being run through a compiler and being translated from your "middle language" or into 0s and 1s the computer can understand. Middleware is code used in between libraries to duplicate their functionality.
https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-is-middleware/
Most original code was written in shell. Most scripting is done in the cli or shell language and stored as a script.shfile, containing instructions to execute tasks. Before python was invented you used the basic shell because nothing else existed yet

load more comments (1 replies)

[–] 7uWqKj@lemmy.world 6 points 1 day ago (1 children)

Bash is perfectly good for what you’re describing.

[–] MajorHavoc@programming.dev 3 points 1 day ago (1 children)

Serious question (as a bash complainer):

Have I missed an amazing bash library for secure database access that justifies a "perfectly good" here?

[–] 7uWqKj@lemmy.world 3 points 1 day ago (1 children)

Every database I know comes with an SQL shell that takes commands from stdin and writes query results to stdout. Remember that "bash" never means bash alone, but all the command line tools from cut via jq to awk and beyond … so, that SQL shell would be what you call "bash library".

[–] MajorHavoc@programming.dev 2 points 1 day ago (1 children)

Thank you. I wasn't thinking about that. That's a great point.

As long as any complex recovery logic fits inside the SQL, itself, I don't have any issue invoking it from bash.

It's when there's complicated follow-up that needs to happen in bash that I get anxious about it, due to past painful experiences.

load more comments (1 replies)

[–] Anticorp@lemmy.world 3 points 1 day ago

I've only ever used bash.

[–] onlinepersona@programming.dev 3 points 1 day ago (3 children)

May I introduce you to rust script? Basically a wrapper to run rust scripts right from the command line. They can access the rust stdlib, crates, and so on, plus do error handling and much more.

Anti Commercial-AI license

[–] Ephera@lemmy.ml 2 points 1 day ago (3 children)

Yeah, sometimes I'll use that just to have the sane control flow of Rust, while still performing most tasks via commands.

You can throw down a function like this to reduce the boilerplate for calling commands:

fn run(command: &str) {
    let status = Command::new("sh")
        .arg("-c")
        .arg(command)
        .status()
        .unwrap();
    assert!(status.success());
}

Then you can just write run("echo 'hello world' > test.txt"); to run your command.

load more comments (3 replies)

[–] NostraDavid@programming.dev 3 points 1 day ago

Basically a wrapper to run rust scripts right from the command line.

Isn't that just Python? :v

[–] Badland9085@lemm.ee 1 points 1 day ago (5 children)

How easily can you start parsing arguments and read env vars? Do people import clap and such to provide support for those sorts of needs?

load more comments (5 replies)

[–] NostraDavid@programming.dev 1 points 1 day ago (1 children)

Can I slap a decorator on a Bash function? I love my @retry(...) (via tenacity, even if it's a bit wordy).

[–] Badland9085@lemm.ee 1 points 1 day ago

I’m going to read this with a big “/s” at the end there xD

load more comments