39

I use both Python (for data analysis ofcourse including numpy and scipy) and R next to each other. However, I use R exclusively to perform data analysis, and Python for more generic programming tasks (e.g. workflow control of a computer model). In terms of basic operations, say operations on arrays and the sort, R and Python + numpy are very comparable. ...


36

Background: I'm a data scientist at a startup in Austin, and I come from grad school (Physics). I use Python day-to-day for data analysis, but use R a bit. I also use C#/.NET and Java (just about daily), I used C++ heavily in grad school. I think the main problem with using Python for numerics (over R) is the size of the user community. Since the language ...


16

(1) What all features should I extract? First, realize that you're not classifying documents. You're classifying (document, query) pairs, so you should extract features that express how well they match. The standard approach in learning to rank is to run the query against various search engine setups (e.g. tf-idf, BM-25, etc.) and then train a model on the ...


15

To answer just your title, yes. Neural nets can give non-boolean answers. For example, neural nets have been used to predict stock market values, which is a numeric answer and thus more than just yes/no. Neural nets are also used in handwriting recognition, in which the output can be one of a whole range of characters - the whole alphabet, the numbers, and ...


15

Joel actually answered this one a few years back. The actual meaning of "teach a machine how to program by itself" is "teach a machine how to take a spec and create a program that corresponds to that spec." And with that in mind: The problem, here, is very fundamental. In order to mechanically prove that a program corresponds to some spec, the spec ...


14

An iterative algorithm is said to converge when, as the iterations proceed, the output gets closer and closer to some specific value. More precisely, no matter how small an error range you choose, if you continue long enough the function will eventually stay within that error range around the final value. In some circumstances, an algorithm will not ...


12

So, I have primarily done data analysis in Matlab, but have done some in Python (and more used Python for general purpose) and also I've started a bit of R. I am going to go against the grain here and suggest you use Python. The reason why is because you are doing data analysis from a Machine Learning perspective, not stats (where R is dominant) or digital ...


11

First of all, it seems to me you are missing (or perhaps misunderstanding) the "explicitly" bit in "without being explicitly programmed" (from the quote in the question). It doesn't mean that no programming is required at all, it means that you are not programming a specific solution to the problem, but instead what you are making is a more general program ...


8

For each class (breed, gender...) of categorical attributes, you can add a number of components to your feature vector equal to the number of possible values in that class. Then, if a data point has the ith value, you set the ith one of those components to 1, and the rest for that attribute to 0. In your example, for gender, you would add two new components ...


8

Is Machine Learning a part of Data Science? No. Big Data vs Data Science Not the same. Birds and Bird Watching are also not the same. Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can ...


7

As an old school (over 50) scientist who has and continues to use a number of these tools I will add my two cents. I have worked with colleagues who still write every piece of code in Fortran, from trivial one-off data analysis jobs to code that dominates some of the worlds supercomputers. Recent Fortran dialects (F90, F95, F2003, F2008) are IMHO, some of ...


7

Usually, such filters are programmed to output not only a yes/no value for each sample, but rather a probability: a sample may, for example, be reported as 95% likely to be abusive. Then you set two thresholds, to divide your results in three groups: very unlikely to be abusive, very likely to be abusive, and uncertain. For example, you might consider ...


7

Work in binary: 0, 1, 10, 11, 100, 101... Know your math: 0+0=00, 0+1=01, 1+0=01, 1+1=10 Know your logic: or, and, not, xor... Find that the low bit is a XOR and the high bit is a AND. Expand the principle of one bit to 8, 16, 32, 64 bits Build it with logic gates. If you want to know more, see my answer to How is fundamental mathematics efficiently ...


7

Fuzzing is a testing method where machine learning can & has been applied. Fuzzing is a method of testing in the realm of automated exploratory testing. It attempts to find defects in software by running a large number of inputs and looking for errors. Unhandled exceptions are the simplest category, but a smart implementation can use ML to find suspect ...


6

You are correct about AI which includes ML which includes DL. NN can indeed be included in ML, be it inside or outside of the DL context. An example for the latter is when neuronal nets are used in simple task based learning (e.g. recognize car number plates in pictures). Data mining is somewhat broader than your definition, because it's not only ...


5

It seems to me that you're comparing apples & oranges. A machine needs 1 day to produce x units, so, this implies that two machines like the first would need 1/2 day. No, it implies 2 machines can produce 2x units in 1 day. We don't have enough information to know if x is divisible, if the machines can cooperate, if there are external constraints (...


5

Yes. This area is hot right now. It's called "big code," and DARPA put $40 million into it: http://www.darpa.mil/program/mining-and-understanding-software-enclaves . Some impressive results have come out of this grant, such as the Prophet and Genesis systems of Fan Long, which can automatically fix bugs in programs by using a learned model of correct patches....


4

Sure, we do this all the time (for extremely limited subsets of problems). It is fairly trivial to imagine taking another step or two and tying something like Siri into the input of these code generators (or something like Wolfram Alpha) which in turn writes code and solves your problem. I would expect that something already exists somewhere to do the most ...


4

The numbers in the table indicate the number of four connected positions which include that space for example: the 3 in the upper left corner is for one each of horizontal, vertical, and diagonal lines of four which can be made with it. the 4 beside it is for two horizontal (one including starting in the corner, one starting on it, one vertical, and one ...


4

Keep a copy of the users' purchasing history within your recommendations service. Then, for each new batch run, it only has to request the updated users from the users service (a delta update, if you want to call it like that). You will need a big update to ramp up the recommendations service, and possibly a big synchronization update from time to time, ...


4

What is convergence, in general The concept of convergence is a well defined mathematical term. It essentially means that "eventually" a sequence of elements get closer and closer to a single value. We call this single value the "limit". The formal definition goes something like this: Given (infinite) sequence of real numbers X0, X1, X2, ... Xn ... we say ...


4

This is a supervised learning problem. It seems to be a classification task i.e. the output variable D takes class labels (the other group is the regression task). There are a lot of algorithms for classification and you should probably start with something simple e.g. logistic regression. If I misunderstood the example and yours is a symbolic regression ...


4

The Coursera course only covers "batch" or "offline" methods for machine learning. In batch methods you train the model once, and then use the trained model as a static resource. What you are looking for are online machine learning methods. Typically this involves finding an algorithm where the new data can be combined with the existing model to generate a ...


4

This is a good candidate for Differential Evolution. DE is a very simple (but powerful) population based, stochastic function minimizer/maximizer. A key point for integrating DE in your scheme is the fitness function: double fitness(Agent_k) fit = 0 repeat M times randomly extract an individual Agent_i (i <> k) switch (result of ...


4

I believe the body of knowledge you are looking for is "Mathematical Programming". In general you want to build a model to support decision making. To do this i would start with a "toy" model. This is where you take a very small example, say you only have one machine and three orders to process. You then need to answer some fundamental questions, like ...


4

Microsoft has been developing DeepCoder to use deep learning to predict a method body from a given input and outputs. That's the only example I know offhand. I can tell you that Meta-Genetic Programming is a field of study with a similar ambition, but I can't say I know enough about it to be knowledgeable. Genetic Programming was in the news in 2015 when ...


4

What you want is often referred to as the strategy pattern. You want to have a strategy for searching, that is easy to swap out. The easiest way to implement this is to accept the search strategy in the constructor of your main class. class MyFavoriteAlgorithm: def __init__(self, observers, search_algorithm): . . self....


4

You're building a training set. This is used to teach the AI what you want. The important thing is to be careful that the set doesn't contain false tells like a red and white checkered table cloth every time it's a pasta dish. We all generalize of course but when humans build training data it's amazingly easy to tip your hand without meaning to. Why ...


4

There are many different types of machine learning. The one in your example is basically trying to draw straight lines between the a and b labelled points. The lines are equivalent to your word mapping and are saved in memory as part of the classifier after you have called the train function. To apply machine learning to your article tagging problem you ...


Only top voted, non community-wiki answers of a minimum length are eligible