Data Science is powerful. By combining the fields of statistics and computer science it allows us to analyze and understand data and make that data understandable to others. This means Data Scientists can direct the public's sightlines to particular trends or information. One particular trend in data science, and STEM in general, worth mentioning is: Despite the growing field, only a small fraction of the STEM workforce consists of minority groups. I'm going to hop right over the messy puddle of why we need more diversity in data science and STEM because plenty has been published on the economic reasons (diversity could help end the tech talent shortage¹), the business reasons (diversity is critical to innovation and productivity²), the feminist and social justice reasons (women and minorities are the most underrepresented in the fields of engineering, computer science and the physical sciences³). In doing this, I'll assume you have made it to the other side with me.

Unfortunately, since Data Science is at the intersection of statistics and computer science, it also brings with it the barriers to entry that women, the LGBT community, genderqueer people, people of color, people with disabilities, and other minority groups face in each of these spaces individually.

Josephine Baker was a scientist in the early 1900's, who was successful in implementing public health practices that dramatically decreased infant mortality in New York's infamous Hell's Kitchen. She was also a queer woman who is wore suits and neckties to succeed in the male-dominated world of public health administration⁴. At times, this resulted in her coworkers forgetting she was a woman and complaining to her about the deficiencies of women doctors.

Today, Silicon Valley is struggling with a culture that arises from being mostly male, mostly white and mostly upper-middle class⁵. When I was job searching, I was at a Data Science networking event where I ran into a senior researcher with whom I was eager to speak. When I approached him, he thought it would be appropriate to ask me if I was thinking of having children. Data scientists have to work to minimize the barriers to entry on an industry, company, and team-wide scale. There is a current tendency to put the onus on an individual to work against barriers in the work place. Books like Sheryl Sandberg's Lean In, for example, can leave the impression that women are impeding their own success by selling themselves short; when, in reality, there are systemic inequalities in place that "leaning in" cannot change.

In order to combat the talent gap surrounding minorities in STEM, I believe recruiters need to work harder to find candidates that are diverse. This means companies have to recruit differently and in different places. For example, a company could decide to foster strategic partnerships with schools that have programs for adults and a good infrastructure for people with disabilities, or look for skills that more closely tie with their future performance than academic credentials.

In my experience, both as a board member of GLOBE and the National Mentorship Chair for Women In Bio, mentorship and networking can make a lasting impact. I had to push myself to be a mentor and be a part of someone's network, especially when I was not sure how I could help. It is imperative that as Data Scientists we foster an analytics-driven culture full of diverse talent, and vocally support someone who is interested in Data Science and how to become a data scientist, especially the next time your team is making a hiring decision.

¹Jery Weinstein, "Could more diversity help end the tech talent shortage?," The Guardian (October 13, 2014). Retrieved from The Guardian: http://gu.com/p/42c43/sbl
²Cristian Dezso and David Ross, "When Women Rank High, Firms Profit," Columbia Business School Ideas at Work (June 13, 2008). Retrieved from Catalyst information Center: http://www.catalyst.org/system/files/why_diversity_matters_catalyst_0.pdf
³National Girls Collaborative Project. (2013, June). Statistics – State of Girls and Women in STEM. Retrieved from National Girls Collaborative Project: http://www.ngcproject.org/statistics
⁴Bert Hansen, "Public Careers and Private Sexuality: Some Gay and Lesbian Lives in the History of Medicine and Public Health," American Journal of Public Health (January 2002). Retrieved from the US National Library of Medicine National Institutes of Health: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1447383/
⁵Nine Burleigh, "What Silicon Valley Thinks of Women," Newsweek (January 28, 2015). Retrieved from Newsweek Business: http://www.newsweek.com/2015/02/06/what-silicon-valley-thinks-women-302821.html

Modern computing has no shortage of tools for the data scientist. The open source community alters the landscape every six to 12 months, and competition keeps you on the bleeding edge. In my career as a data scientist, I use everything from scientific Python™ packages to the newest cloud computing architectures—and sometimes all within the same project, as the initial stages of data exploration and mining are often done in a different language than the final product implementation. Here is a brief tour of my experiences with some essential tools for data science:

Python - Python has the richest collection of packages I have come across. When I see a new data set, I am inclined to tackle my problem by dissecting the data with one of the scientific libraries (SciPy, scikit-learn, etc.) and then visualizing the results (Matplotlib). Python is the easiest tool to use and has provided me with the most mileage for my data set investigations.

Java™ - Java is the backbone of cloud computing and a must-know for any data scientist who wants to build portable, production-quality products.

Scala - This language acts as a flavor of Java and is an easy, functional, programming language. Scala is the natural language for the latest distributed computing platform, Spark. As a high-level language, Scala allows the author to focus on what should be done with the data set and not on how to position the data to do it effectively. This is one of my favorite tools.

Hadoop® - It is the quintessential cloud environment. Hadoop provides long-term storage of data across a cluster of computers.

Storm - Originally developed at Twitter, this tool enables stream processing of data collected from live feeds. This is an easy tool for Java developers working on streaming analytics projects.

Spark™ - This replaced the old MapReduce paradigm by hiding the processing details behind the functional language Scala. It has the ability to work with streaming data and with large static data sets. This is the latest and greatest tool in distributed computing.

HBase - A sturdy Hadoop-based database system that is easy to use.

Kafka - This message passing system sits between the raw data source and the consuming process like Spark or Storm. Kafka prevents data loss between the source and the analytics engine in the streaming analytics process.

Mahout™ - A configurable cloud analytics toolkit that allows for advanced computing techniques through an API, allowing the data scientist to think about the data set instead of the code.

Elasticsearch - The latest search engine platform on top of which you can place a friendly Kibana user interface.

Lucene™ - This tool is the backbone to open source text processing—and the basis for text processing in many other tools, including Mahout, Solr™, and Elasticsearch. This is a must-have for text processing gurus.

I hope you will find these tools useful in examining the data sets in the National Data Science Bowl and in your everyday life as a data scientist. Good luck!

Overfitting is an issue within machine learning and statistics. It occurs when we build models that closely explain a training data set, but fail to generalize when applied to other data sets. Overfitting is a part of life as a data scientist. We all do it to some degree or another. In the case of forecasting in data science competitions, it might actually be advantageous to overfit to Kaggle's public leaderboard. However, if you have an independent, and identically distributed (iid), split between train and test data sets, then it's probably better to come up with a leak-free cross-validation (CV) scheme.

Preventing leakage—the creation of unexpected additional information in the training data set resulting in a machine learning algorithm making unrealistically good predictions—is actually a difficult task and typical k-fold or random split may not always prevent leakage. You may need to use some kind of stratified sampling scheme. Given Kaggle's typical allowance of two entries, you may want to select one entry that maximizes local CV scores and one that maximizes public leaderboard.

When thinking about overfitting, here are some points about the National Data Science Bowl to keep in mind:

Am I making multiple comparison corrections on my score given a large number of submissions? As a general rule, you should expect a public to private ranking drop proportional to the number of submissions (rank) unless you follow a very rigid CV scheme.

Do I want to overfit on purpose for ensembling gains? Occasionally an overfit model can work well within a large data set. This is sometimes seen within neural net ensembles. The best individual scores do not always make for the best ensemble marginal gains. Note: this does not mean you want the final ensemble model to overfit. Also diversity via surrogate loss functions can sometimes add value via underfitting. You could try hinge loss instead of logistic loss. Ensembling many diverse models can help mitigate overfitting in some cases. Strategies for ensembling can involve bagging from the row data, use of different algorithms, or trying different groups of features.

If the National Data Science Bowl allows semi-supervised learning, in theory leveraging more data is always better given infinite AWS computational resources. However, in practice, these methods have not always produced champions. Consider it an open challenge.

You can bound the error given iid data train/public test/private test splits and fixed sample sizes. So, overfitting in this case is not a bad idea when the number of test set rows (observations) is very large (in the billions) and the number of columns (features) is less than the number of rows. Small train and test sets will require the data scientist to become more cautious and more vigilant with CV.

It is good to know exactly how the public to private data split was made, and exactly how the train to test data split was made. However, this might not always be available for every data science competition. The best way to avoid overfitting in data science is to only make a single Kaggle entry based upon local CV. Tim Salimans obtained second place in "Don't Overfit!" with a single submission. However, it's often more fun to grind your way into a stochastic (public) leaderboard descent. I suggest you explore and find options that work best for you. The leaderboard is temporary (even in the National Data Science Bowl), but the data science knowledge you gain can last a lifetime.

There has been a lot of news coverage lately around the topic of creating a data-driven culture within an organization. The fact of the matter is a data-driven culture is crippling. We tried to create a data-driven culture too, but ultimately found that our real transformation came by using data as inputs into a real Analytics driven-culture. A culture that values true experimentation, understands failure is the price of discovery, and actually makes use of analytic outputs for decision-making.

While there is common agreement about the power and potential of data science and advanced analytics, many organizations struggle with how to incubate and sustain a curious and innovative Analytics-driven culture. It is a concept that continues to perplex organizational leaders and prevents organizations from reaching the next level of performance. While often viewed as an organizational problem to be addressed by senior leaders, we have seen first-hand how every data scientist in an organization has an important role to play. Leadership from the top is critical, but the support of all data scientists is needed to create a cultural change.

As Booz Allen Hamilton formed its 500+ data science team, we faced some challenges in creating our own Analytics-driven culture. In the spirit of sharing, I wanted to share a few key practices (and some lessons learned) from our own experiences. These practices helped us sprout and grow analytics as a central cultural tenet.

1) Failure is the Price of Discovery. One of the first things we had to do on our journey to an Analytics-driven culture was recalibrate our tolerance for failure. To realize the full power of what data science and advanced analytics can provide, we needed to create conditions that foster curiosity and experimentation and embrace failure for the unexpected insights and learning that it can provide. At Booz Allen, we created these conditions in a few ways. First, we empowered our data science team to investigate and discover; to design and run their own experiments that blend inductive pattern recognition with deductive hypothesis generation; and to explore "rabbit holes" of interest. Recognizing, like many others, that data scientists are rarely motivated by money and prefer the time and space to "geek out", we established a series of reward and recognition mechanisms focused on investing in passion projects. What we learned in our journey is that true curiosity and experimentation requires an enormous tent of skills, abilities, and perspectives to achieve meaningful outcomes for data science. It is important to take an inclusive approach to advanced analytics that extends beyond the traditional cadre of world-class data scientists. For example, our data science teams often include organizational design and strategy practitioners, design thinkers, and human capital planners. Finally, discovery has an added organizational benefit. We found that discovery is self-fulfilling and that once you start on the road to discovery, you gain momentum and energy that permeates throughout the organization, which provides powerful lift to your data science team.

2) Story Tellers Learn by Doing. Data science and advanced analytics can be intimidating for some and exciting for others. What's important is to provide a variety of learning opportunities that can cater to the different skill sets and attitudes that exist within the organization. We tackled this challenge in a variety of ways: 1) We sponsored analytical challenges and hackathons to both test and satisfy the desire of advanced practitioners while at the same time providing a safe learning environment for those still developing their skills; 2) We created a self-paced online training course for data education, Explore Data Science, to teach both foundational and advanced data science concepts such as data visualization; 3) Additionally, we created a data science training program called TechTank to cultivate skills through apprenticeship and structured training. These are just a few examples. The bottom line is that we learned it is important to design opportunities and learning programs at all levels to help people get familiar with data science concepts and become more comfortable in practicing it. Don't cast data scientists as mythical rock stars and then not offer a way into the "club". Remember, all data scientists have a role in providing mentorship and learning opportunities. Attend hackathons, teach classes, and mentor junior staff whenever possible.

3) Adjust Your Time Scale. One of the key benefits that we have found from our Analytics-driven culture is the difference in timescales in findings solutions. With an Analytics-driven culture, the organization becomes particularly adept at asking the right questions and rapidly (we're talking minutes here) getting answers. That shifts the time available to thoughtful discussion on what analytical outputs mean and how they inform decisions to derive desired outcomes. For example, at one government agency, our Analytics-driven perspective led us to abandon a traditional data-driven approach, such as clustering models, in favor of developing models of ‘normal' behavior and then filtering the data set to decrease the signal-to-noise ratio in order to catch fraud in a significantly shorter amount of time. This pioneering technique was a direct result of data scientists thriving in a culture that values discovery and ingenuity.

While these are just a few examples of how we have approached our journey to an Analytics-driven culture and the benefits we have received, we hope that the ideas presented here inspire you and offer some practical ways to help you get started on your journey to an analytics-driven culture. Remember that everyone in the organization has a role to play from organizational leaders to data scientists. It's through the collective effort of these groups that you can achieve an Analytics-driven culture.

Are we making sense? Check out more of our perspective at: www.boozallen.com/consulting/strategic-innovation/nextgen-analytics-data-science

Support Vector Machines (SVMs) may not be as popular as Neural Networks within data science, but they act as powerful, useful algorithms. One of the difficulties of SVMs has been the computational effort required to train them. However LIBSVM, which has been used for over a decade, can fairly easily handle the 30,000 training points in the National Data Science Bowl competition's data set. That makes SVMs a viable tool for you to use both in general, and for the purpose of competing in the National Data Science Bowl.

At their heart, SVMs find a linear solution to the problem presented from the data set, similar to the long-used Logistic Regression and Linear Regression. The notion behind what the SVM does differently is that it tries to maximize the distance between the classes involved. This is also known as margin maximization. The intuition being: If you find the line that maximizes the distance between two classes, it's more likely to generalize well to unseen data.

What makes SVMs in data science truly interesting is what is known as the "kernel trick." This allows one to have the algorithm transform your input features into a different feature space and find a linear solution, which may be non-linear in the original dimension. The form of a kernel trick is K(x,y) where x and y are two features, and it outputs a single, real value where larger values indicate similar and smaller values with less similarity between x and y. For example, if you wanted to include the interactions of every combination of two features, you might create extra features that represent the multiplication of those features together. This is equivalent to creating the degree-2 polynomial of the features, which the SVM can do as a kernel trick. However, since the SVM can do this without actually forming the features, you can raise the degree as high as you like and it will still work (if you tried this yourself you may very quickly find yourself unable to learn to model or even fit the data in memory).

Another commonly used kernel is the RBF kernel, and is probably the most widely used kernel. It has the nice property that using it with an SVM can be interpreted as a type of smarter nearest-neighbor search.

Continually, the SVM combined with Kernels have some very nice mathematical properties—one of which is you can add or multiply kernels together to get a new, valid kernel. We can use this to incorporate different types of features more elegantly into a single model.

In the case of the National Data Science Bowl data set, we could use a combination of three kernels—one for each feature type as Aaron Sander (one of my fellow data scientists) suggested in his post (say k₁, k₂, and k₃). If I noticed spatial features (k₁) tend to only give low scores when two inputs were definitely different classes, I could define K(x,y) = k₁(k₂ + k₃). That way, when spatial features indicates a low match, it will strongly discourage the algorithm—even if the other set of features thought it was viable. I could even add some extra knobs to tune, making K(x,y) = k₁(c₂ k₂ + c₃ k₃) allowing me to favor one set of features as better than the others.

This type of feature combination is especially useful for a data set that cannot be represented as a fixed-length feature vector. Kernels can be defined directly for text data, graphs (like the connections in a social network), and lots of various structured problems. By using a different kernel for each feature, we can use SVMs on more types of information than other algorithms can, and we can use them all simultaneously.

Hopefully, this has motivated you to explore the SVM as a potential option for you to solve this, and future, problems using a complex data set. For more information about the LIBSVM, the authors have a short guide on how to use their software, which includes good advice on using SVMs in general. It's easily available in python (scikit-learn), R (the e1071 package), Java (Weka), and has been ported on its own and wrapped into numerous other programming languages and libraries as well.

Feel free to talk or ask me questions @EdwardRaffML. Good luck!

On January 15-16, 2015, Hatfield Marine Science Center researchers hosted data scientists from Booz Allen Hamilton and Kaggle for a site visit.

We were incredibly excited to finally meet our data science partners from Booz Allen and Kaggle face-to-face during their site visit on January 15-16, 2015. We had communicated for months prior to their visit, discussing data, analytics, the number of plankton classes for the data science competition, and the general ins and outs of the National Data Science Bowl (http://www.datasciencebowl.com/). Unfortunately, the weather didn't seem to realize how momentous this meeting was. The previous week, we had beautiful blue skies, 60-degree weather (pretty warm for Oregon in January!), and planned many outdoor excursions for our visiting data scientists. However, on the actual day of visit, the clouds rolled back in and the rain decided to start blowing sideways. At least they experienced classic Oregon weather!

Nonetheless, like the ultimate professionals these data scientists are, our visitors from Booz Allen and Kaggle were receptive and enthusiastic throughout the visit; this included a tour of the Research Vessel OCEANUS, presentations from our group, and tours of multiple labs at Hatfield as well as the O.H. Tinsdale Wave Research Lab and the Ocean Observing Initiative at Oregon State University (OSU), Corvallis.

We wanted to show the breadth of marine science research being performed at OSU and how scientists were dealing with "big data" and how data can support social good in their work. We hoped that this visit would highlight the different fields where data science could be used to push the frontiers and make "waves" in our marine research. We showed our visiting data scientists how big data was a challenge scientists studying whale migrations an salmon genomics, as well as those operating ocean observing platforms comprised of moored instrument arrays and autonomous underwater vehicles, were trying to tackle.

We were impressed by the diversity of backgrounds in our visitors and how they loved to use data science to solve problems of all sizes, from big (e.g., classifying plankton imagery) to small (e.g., the seating arrangements of a wedding). This passion was evident in their enthusiasm while meeting with Dr. Thomas Dietterich, whose research in machine learning is notable for its application to ecological data sets like crowd-sourced bird counts.

We hope that our Booz Allen and Kaggle partners came away from this visit with a deeper understanding of how the outcome of the National Data Science Bowl competition will benefit marine science research and, in turn, social good.

We look forward to seeing the outcomes of this competition and for our data science problem to be solved!

Building on Paul Yacci's earlier post on the importance of feature selection in data science and data analysis, the creation of new features from your existing data set can play a large role in the performance of your model in data science. There are multiple methods of feature creation and data transformation. Often, finding the right transformation of your data can reveal relationships that would be difficult see otherwise, and may also make it easier for your model to separate classes. In the simple case of creating linear models for regression, this can take the form of squaring terms, taking their logarithm, or other functional transformations that bring the fundamental problem back into a linear relationship through the transformation. For image data in particular, there are a number of useful transformations in data analysis.

1) Spatial morphology. The shape and structure of objects plays a huge role in how humans and your models can classify objects. Are the objects of interest elliptical? How many segments do the animal have? Does it have a tail? How solid is the image (i.e. is it solid or filled)?

In the case of plankton data analysis, it can be useful to explore the ways that experts classify the images of plankton (see the plankton portal field guide and try hand classifying a few images to get a feel for the data). Also, don't forget to look out for strange creatures like the siphonophores that may defy simple morphological classification.

2) Fourier and Wavelet Transformations. These transformations change the problem from the spatial domain into the frequency domain and, in the case of wavelet transformations, allow you to trade off between the two, revealing signatures that might not have been visible otherwise. Using either the power spectra or selective wavelet coefficients can allow you to capture some of the information about how much structure is present in the images (see the popular practical guide for wavelet analysis and this example for wavelets being applied to describe structure in overhead imagery).

3) Feature Space Descriptors. SIFT or (Scale invariant feature transformations) or more recent algorithms such as SURF (Speeded-Up Robust Features), RIFT, etc. are used to describe features around points of interest in images by creating vector representations of the differences of image intensity at multiple scales. The collections of these features can be clustered and quantized in a bag of visual words method to classify images. The feature descriptors alone can also be a good proxy for texture.

Data Science was named Forbes Magazine's sexiest profession of 2014 as well as being the most trending STEM career in Ebony Magazine's July issue. This has led to many wondering how they, too, can enter the data science profession. So, what is a typical day in the life of a data scientist?

How is data science similar to other jobs?

"A data scientist is someone who is better at statistics than any software engineer and better at software engineering than any statistician."
- Josh Wills of Cloudera

The above quote illustrates two jobs closely related to data science and the hybrid nature of big data analysis. Generally, statisticians are not apt in trending programming languages, data structures, or the advanced areas of computer science and many software engineers may not know as much about advanced mathematical and machine learning algorithms for analyzing, clustering and classifying data. The newness of the term "data scientist" has led to many disagreements on an formal definition of the term, but speaking with many in the field a typical work day is generally centered around the concept of writing code for mathematical and statistical algorithms, often dealing with "big" data sets.

What kinds of problems does a data scientist solve?

At a basic level, the general task of a data scientist is to search for patterns in large data sets. There is generally a lot of context left out of this task, however. Once we have an understanding of the data and what questions we would like to answer, there is a question of the best and most efficient methods to answer these questions. These two do not always agree and when they do not, the difference can often be grand. This is a common problem in computer science. For example, when we're searching for the maximal element, do we need to sort all the elements and select the element at the top of the stack, or simply run a search algorithm. Similarly in data science, a question may be whether there is a need to run an algorithm such as clustering (which may be expensive if it calls for multiple iterations through a large data set) or can we answer these questions with a simpler distance calculation?

How much of being a data scientist is sitting in front of a computer vs giving presentations vs working in groups?

Most of my time as a data scientist is spent researching, writing algorithms and writing code to answer the questions about the data sets in question. A fundamental part of data science involves group work - obtaining the data, understanding the data, and understanding and analyzing what is wanted from the data. Whether all these roles are filled by one person (me) or by a group of people depends on how your team is set up, but as the data scientist I generally have someone I am working with who can provide me with more insight on the data, answer relevant questions and clear up any confusion. In addition to this, there may be questions from developers and statisticians about our work as a data scientist that compare the methods we used to other methods in their fields. In these type of settings, it may be important for a data scientist to give some presentations to be able to answer such questions.

What is the most stressful/rewarding thing about being a data scientist?

As a mathematician there is a certain feeling of satisfaction in seeing the need for advanced algorithms to help solve problems in the real world. It's one thing to read a textbook with example problems. It's a totally different feeling to hear about a real world problem and use your knowledge to solve it. Similarly, being able to write the code and see this develop from a problem, to an idea to an algorithm, to a running program is a great and enjoyable process.

However in order to reach that last stage of a running program, we often have to go through what's known as debugging. This is a process of searching for errors in code that either prevents the program from running or leads to incorrect solutions. This can be a challenging process for a number of reasons - some similar to the reasons a developer would find debugging stressful, and others because data science often involves working in cloud environments which require some of the standard practices for debugging programs in a traditional environment to be revised.

Feature Selection is crucial to any model construction in data science. Focusing on the most important, relevant features will help any data scientist design a better model and accelerate outcomes.

So what exactly is a feature in data science analysis? Let's start there. Frequently we refer to the set of values that describe a data point as features. In actual practice, these values can go by multiple names, attributes, values, and dimensions. However, they refer to the same thing: measurements that describe an instance.

Here's an example.
To describe a single day of the year, what kinds of measurements could we gather?

Some values that you might encounter:

• Temperature
• Air pressure
• Hours of sunlight
• Precipitation
• Humidity
• Number of pennies that were found on the ground that day
• The number of jaywalkers that day in New York
• Price of oil
• Economic jobs report released today

If we were interested in predicting the weather the next day, which of these measurements would be most valuable? Most likely, the temperature, air pressure, hours of sunlight, precipitation and humidity would yield the best prediction.

Data regarding pennies, jaywalkers, oil and economic jobs would likely not yield any useful information regarding the weather. However, it is always possible that there might be some form of relationship that exists mathematically, although it is not explainable. (This is referred to as a spurious relationship in statistics.)

If we dedicate resources in our learning algorithm to uninformative features such as these, we complicate our algorithm. And that ups the likelihood that we will overfit the model to our dataset and thus learn incorrect concepts. My example was simplistic. Now imagine there were 1,000 more features, like socks color of a random 1,000 people. This data won't help predict the weather. But if it's in the dataset, the algorithm would waste huge effort configuring and tuning parameters trying to learn the relationship between socks and weather the next day. There's no value in that. If we only had a few parameters, we could make much more efficient use of the algorithm/model. So discovering a small set of powerful features is the goal.

There is another case where one feature carries the same information contained in another feature. In my example, a redundant feature may be the presence of precipitation and the amount of rainfall that day. The amount of precipitation is more detailed than a simple "yes/no" value. A model trained on this data will spend extra effort learning values that it already has information on.

Ideally we would like to have a set of features that have relevance to the model being constructed and carry unique information amongst each other.

Feature selection is critical in situations where the number of features greatly outnumbers the number of samples. This is referred to as The Curse Of Dimensionality, a rich topic for another time. For a quick overview:
http://en.wikipedia.org/wiki/Curse_of_dimensionality.

The basics are that you need an increasingly larger number of samples to cover the instance space at the same density. A helpful interactive demo of this concept can be found here https://prpatil.shinyapps.io/cod_app/

Now that we understand what a feature is and why too many can be a bad thing, what can we do?

Two common approaches for feature selection are Filter Methods and Wrapper Methods. In both methods, features are evaluated to assess the quality of a model that could be constructed from this feature set. A filter method generally looks at features independently, evaluating the relevance of each particular feature. A filter method would score the features independently of how they perform on the model of interest. For example, a simple filter method that could be used is a t-test. This looks for significant differences between classes, looking only at one feature at a time. We could go through each feature one by one, testing for significance between the class distributions, keeping only the features that are significantly different.

A wrapper method evaluates the features in relation to their performance on the model. The set of features are used to construct the model and the performance of the set is scored. Feature sets that perform better are indicative of good feature sets. You can think of the feature sets as a team – and the team is evaluated for its overall quality as compared to another "team" of features. For the weather example, if we selected temperature, air pressure, and precipitation and constructed a predictive model using only these features, how well would it predict tomorrow's weather? This result could be compared to a model constructed from jaywalkers and the price of oil. The feature set containing weather measurements would likely have a better model accuracy.

What makes a good feature set? Everything depends on what you want to accomplish with your model. If you ask different questions of your data, a feature could have different importance.

The next time you face a large set of features with your dataset, consider whether the presence of all of them will be beneficial. If not, try filtering or wrapping.

Let's continue the conversation @paulyacci

Being a data scientist is more than having a technical background; it's also about going beyond your tools and understanding what it really means to tackle complex data analysis problems. No matter if you are a seasoned big data expert or are just considering moving into the field, here are five things you ought to know.

• 1. Assuming the title, Data Scientist, does not make you one. There are lots of individuals that claim the title without any robust experience with big data or data analysis. However, there are likely as many people who have the experience needed to be a 'data scientist' but do not claim the title. Many disciplines require DS skills to understand data; the only major difference between a physicist and a data scientist is that the latter applies the math/cs skills to many types of data with the assistance of a domain expert. The former is also the domain expert in a specific area of physics.
• 2. Speed is sexy, live in the fast lane. Technology and resources for data science are in constant flux. Being able to stay on top of the change is vital. Recently, Spark has tromped Map Reduce, drastically changing the "hottest tool" for big data science. If you fight the change, you will be left in the dust.
• 3. Tools are just that, tools. Having access to machine learning libraries does not make a data scientist. Know the tools, use the tools, but most of all think about the problem. Depending on the situation, you may not have tools or you may need to approach feature extraction from an 'outside of the box' way. To be successful, you may need to turn your data science problem on its head.
• 4. Value your domain expert. Domain expertise is vital to your success and is not something you can pick up overnight. Many times the domain expert loaned to you for your work has been studying the data for years or even decades. Treat them with the respect they deserve.
• 5. Know yourself. Take note of your strengths and weaknesses. When you have a data science team where you cannot fill a gap due to a weakness, own that. Acknowledging that you need to reach out to colleagues or read academic papers is the first step toward defeating any knowledge gap. Chances are you can learn what's missing -- just give yourself the time to do that.

I'm often asked "what are good things to do to get involved with data science?" In this post I'll share with you some key activities you can get started with today.

Whether it is photos on phones, links in a social network, or information around health, our society has an increased appreciation and enthusiasm about data. It's an exciting time as open data initiatives are met with open software tools that enable large-scale analysis. To actually get insight from data requires intellectual curiosity and a mix of skills and knowledge.

Learn more math

Being able to formalize different relationships in data enables you to identify potentially useful features. Statistics plays an important role for testing whether candidate relationships are salient or spurious. Datasets are often high-dimensional and are increasingly large. The ability to think of subspaces and projections in data is an important aspect of data exploration. Finally, building a deep understanding of the advanced algorithms used by data scientists (e.g., machine learning, regression) require a solid foundation in the underlying mathematics.

There are a variety of ways to expand your knowledge of mathematics. Here are just a few:

• University courses – These are a great way to build foundational knowledge in topics such as statistics and linear algebra. Check out your local university and community college for the courses that are available.
• Online courses – Topics covered range from basic mathematics to advanced machine learning. Sites like Coursera, Udacity, and Stanford Online offer a variety of courses in math, statistics, and machine learning. Sites like Project Euler helps you to build problem-solving skills in addition to mathematical insights.
• Self-study – Never discount the classic method of purchasing a textbook and working through problems. This can be a great way to bolster skills around a particular topic or concept. Plus, textbooks are cheap now!
• Blog posts and tutorials – There are a variety of great blogs available with illustrative tutorials around complex mathematical topics. One of our favorites is LEARNING Lover.

Build things

Data scientists build all sorts of things from analytic pipelines in order to set up predictive models to data products such as recommendation engines. Necessarily, computer programming is critical to processing and systematically analyzing data. We are often asked what programming language should I learn for data scientists. There is always lots of debate around this point, but you can't go wrong with either R or Python. Scala has also become popular as data scientists are increasingly using Spark.

There are a huge number of online resources to help you get started. CodeCademy and Code School have free courses on Python and R respectively. Individual open source projects also have documentation and tutorials you can reference. Explore Data Science has a great sequence of modules interweaving key math concepts and Python programming.

Coding during hack-a-thons and meetup events is a great way to learn from others. It also fosters collaboration, a vital activity for data scientists. By getting out in the community, you'll begin to grapple with challenges in uncharted problem spaces where innovative solutions are needed. NASA's space apps challenge is a great example. There are also organizations like Data Kind who host hack-a-thons that connect data scientists to social good problems. Hack-a-thons can also lead to projects requiring a sustained coding effort, which can be invaluable in enhancing your technical acumen.

Be a data ninja

The ability to slice and dice data enables data scientists to run down hypotheses. Data scientists generate hypotheses both through data-driven approaches as well as from leveraging domain knowledge. The key characteristic of a data ninja is agility, and it's upon this ability that the effectiveness of data scientists hinges. In this way, you can search for insight in datasets in an interactive way, where the testing of one hypothesis may lead to another. This is where the real art of data science comes into play.

The best way to learn these skills is by doing. Actively seek challenging problems to solve. Data science competitions from sites like Kaggle provide "prepackaged" problems and data sets. Simply exploring open source data sets from places like Data.gov can help build valuable skills. Don't overlook the hack-a-thons and meetup groups previously mentioned in this post. There are data science meetup groups in nearly every major city now, providing great resources to help connect you with challenging problems. They have the added benefit of connecting you with the data science community.

This post doesn't touch on everything you need to know to be a data scientist nor every method of learning. It should, however, help get you started on your journey to becoming a data scientist. As you explore these three key areas you'll quickly identify knowledge gaps that you can explore further. Never stop learning! If you're building out experience in math and coding across different domains, you're well on your way.

Links

• www.coursera.org
• www.udacity.com
• online.stanford.edu
• projecteuler.net
• learninglover.com
• www.codecademy.com/tracks/python
• www.codeschool.com/courses/try-r
• www.scala-lang.org/documentation/
• www.python.org/doc/
• exploredatascience.com
• 2014.spaceappschallenge.org/challenge/
• www.datakind.org
• www.kaggle.com