4 stars based on
But between the pretty text of the law and what really goes on in the intestines of the bureaucracy there is a huge gap. Every government agency is in favor of data sharing thiago binare optionence except when it comes to its own data.
The actual reasons are not always easy to pinpoint. Sometimes there are legitimate concerns about privacy, as in the case of fiscal data. But this is rare. More often than not the thiago binare optionence privacy concerns are just a convenient excuse.
In some cases the privacy excuse is used even when the data is already public. For instance, everything the government buys other than, say, spy gear goes in the official bulletinwhich anyone can read. The equivalent of the IRS in Brazil - Receita Federal - has all the corresponding tax invoices neatly collected and organized in a database.
That database would make it much easier to compute, say, the average price the Brazilian government pays for ballpoint pens. Government agencies have been trying - and failing - to put their hands in that database for years. But the law says that government purchases must be public.
The data is already public thiago binare optionence not in a machine-readable format. What if my data shows that I have way more people than I need? In general the more an agency refuses to share its data the more important it is to get it. People sometimes assume that our asking for their data is the prelude of bad things. We need to explain that inside the Office of the Comptroller-General there is this little unit called the Observatory of Public Spending and that our goal at the OPS is often to help other agencies by extracting insights from their data.
So you end up guessing based on the contents themselves. As you might thiago binare optionence, sometimes you guess wrong. Thiago binare optionence mistake an ID field for a numeric field. Besides the lack of documentation there are also many errors. The problem begins in the data generation process, i. These systems are too permissive; they lack basic validation like checking input type numeric, text, date, etcinput length does the state code have more than two characters?
And there is no punishment for the bureaucrat who enters incorrect data. The most frequent result is absurd averages. You try to compute, say, spending per employee, and what you get back is a bazillion dollars thiago binare optionence something close to zero. That means many hours of data cleaning before we can really get started. The Brazilian government does not deliberately recruit data scientists. Data scientists come in through a bunch of miscellaneous doors and then we find each other - by word-of-mouth or Twitter or conferences - and come up with ways to work together.
In order to enter the Brazilian government you usually have to pass a public exam and such exams do not cover any content even remotely related to data.
What you do need to learn to pass these exams is a large number of arcane pieces of legislation, mostly relating to government procedures - like the three phases of a procurement process, what they are called, what the deadlines are, what the many exceptions are, and so on. That makes recruiting for your team a lot harder than in the private sector, where you can simply post a job ad on LinkedIN and wait for applications.
It goes more or less like this: You start by identifying the person you want to bring to your team. He or she will usually be in another government agency, not in your own. You will need to negotiate with his or her agency so that they okay their coming to work for you.
That may require you to find someone in your agency willing to go work for them. If the head of your agency your minister or whatever has sufficient political clout and know that you existthiago binare optionence or she may try to prevail over his or her counterpart at the other agency. They need to tell their current agency that they no longer want to work there, but they have no guarantee that they will get transferred. Imagine telling your significant other that you want to break up but then being somehow legally compelled to stay in the relationship.
It can be awkward. They may start giving thiago binare optionence less important tasks. Such possibilities often kill any recruitment before it even begins. There is also the issue thiago binare optionence salary negotiations. The issue being that they are not possible. When you work for the government the law determines your salary - there is no room for negotiation.
So whoever you are trying to lure must be really excited by the work you and your team do because the work itself is pretty much all you have to offer. Paradoxically, in the midst of this red tape jungle we have a lot of leeway to play around thiago binare optionence try new things.
Data science is a highly technical subject, one that takes at least a few Coursera courses to even begin to grasp, and that helps keep it unregulated. We have to thiago binare optionence out the same stupid forms everyone else does when we want to buy stuff, but whether we use Hadoop thiago binare optionence not, whether we adopt Python or R for a project, whether we go with an SVM or a neural network or both, and whether we think any given project is worth pursuing is all entirely up to us.
There is also the tenure factor. You see, in Brazil once you enter the civil service you get automatically tenured after three years. And the constitution says that, once tenured, you need to do something really outrageous to get fired - and thiago binare optionence then there are several appeal instances and often times nothing happens in the end. It takes something thiago binare optionence criminal to get a government employee fired. Like, they need to catch you taking a bribe or thiago binare optionence showing up for work for months.
Overall thiago binare optionence is bad: But for experimenting purposes tenure is great. Some bureaucrats want our assistance and happily give us their data and collaborate with us, helping us understand their problems and needs.
And they worry that you might disrupt the way they work or that you might automate them altogether. Without tenure heads might roll. Most of that money goes down the toilet: But data science can help. That way the government will be alerted of potential problems before thiago binare optionence signs the contract.
If thiago binare optionence do academic research but are not affiliated with an academic institution you probably know Sci-Hub. It gives you access to over 60 million research papers - for free no ads, no malware, no scams. Alexandra Elbakyan, its creator, has deservedly been ranked by Nature one of the top ten most relevant people in science and we independent researchers owe her a lot.
In the Brazilian government, where I work, Sci-Hub is not only not welcomed, it thiago binare optionence actively blocked. So I thiago binare optionence to build an interface to Sci-Hub - an app that takes my search string, gives thiago binare optionence to Sci-Hub, and retrieves the results. Much like I did before in order to use Telegram. I wrote it on Thursday evening and I was confident that the next morning I would just fire app a new project on Google App Engine, deploy the code, and be done with it in less than thiago binare optionence hour.
I ended up working on it all Friday and all Saturday morning; only at Saturday What follows is an account of those 36 hours, largely for thiago binare optionence own benefit in case I run into the same issues again in the future, but also in case it may be helpful to other people also looking to unblock Sci-Hub.
All you have to do is send a POST request. With email I can have a background process do the heavy work; that way I can send several POST requests in a row without having to wait in-between.
I wrote two scripts. This is what happens when data scientists do web development. The other script is the backend. It is launched by the frontend with a call to subprocess. That way all requests are independent and run on separate background processes. Both scripts combined had lines of code. Code written and tested, Thiago binare optionence turned to Google App Engine for hosting the app.
With only lines of code and two scripts I thought that launching the app would be a breeze. GAE Launcher was a nifty thiago binare optionence app that made deployment really easy. I had been using it since and it allowed me to focus on my app and not on deployment nonsense.
It thiago binare optionence some half an hour of googling and debugging before I could get it to work. And the flexible environment is a different ball game. I had never used the flexible environment before I think it only became generally available early this yearbut I decided to give it a try. The exact same code that works fine on my machine returns a mysterious Application startup error when I try to deploy the app.
Thiago binare optionence deploy attempt generates a log file but it is equally uninformative, it only says Deployment failed. Thiago binare optionence to cleanup deployment artifacts. I declared all my dependencies in my requirements. That resulted in from thiago binare optionence import BeautifulSoup raising ImportError: No module named html.
After several unsuccessful attempts to point pip install to a specific source file I gave up on pip. That did the trick. Popenwhich is a call to the OS. It looks great and one day I want to sit down and learn how to use it. But by the time I got to this point I was entering the wee hours of Saturday. My hopes of getting it all done on Friday were long gone and I just wanted a quick fix that would let me go thiago binare optionence bed.