Your Algo's Backtest Is a Sample. I Ran the Test Nobody Shows You.

I recently finished HKUST's Python and Statistics for Financial Analysis on Coursera. It teaches you to pull stock data into Python and actually test it: confidence intervals, hypothesis tests, regressions. Basically, the stuff you're supposed to do before you believe a chart.

If you've spent any time on Indian finance Twitter or YouTube, you've seen the screenshot. Someone's algo backtest. Big green CAGR, high win rate, tiny drawdown. "Built this in 10 minutes, no coding needed."

So I wondered: if I take a popular strategy and put it through what the course taught me, does the edge survive?

Short answer: no. The backtest looks great. The statistics say its whole "edge" over ten years basically comes from one lucky quarter in 2020.

1. SEBI now tracks the algo. Not whether it works.

India just rewrote the rules for retail algo trading, and none of the new rules ask whether your backtest actually means anything.

It started with SEBI's 4 February 2025 circular on "safer participation of retail investors in algorithmic trading". Brokers and algo platforms pushed back, so a 30 September 2025 extension gave everyone a step-by-step timeline:

Now every algo order carries an exchange-assigned ID, and your broker is responsible for every algo running on its platform. Honestly, that's a good fix. Before this, nobody could reliably say which strategy fired which order.

But look at the scale. In January 2024, Zerodha opened Streak to all its users, backtesting included. By the end of FY26, Zerodha had 68.94 lakh active clients. That's about 69 lakh people one click away from a backtest that spits out a CAGR, a win rate and a drawdown.

SEBI can now trace which of those backtests turned into a real order. It can't tell you if any of them were better than luck. So I built one to find out.

2. Building the strategy

On paper, this strategy looks like something people would happily pay to subscribe to.

Module 1 of the course builds a moving average strategy on a US stock. I rebuilt it on the Nifty 50. The rule is really simple:

● If the 10-day average price is above the 50-day average, be fully invested tomorrow.

● If not, sit in cash.

That's it. This kind of rule-based crossover is exactly what no-code algo platforms are built for.

One small detail matters a lot. The signal is worked out at today's closing price, but the trade only happens tomorrow (that's the shift(1) in the code). If you skip that, your backtest is basically using prices it couldn't have known yet. That's the easiest way to fake an edge without realising it.

I ran it on ten years of data, from 4 January 2016 to 31 December 2025. That's 2,464 trading days, including the Covid crash, the messy 2022 rate hikes and the 2024-25 correction. Here's what a marketplace listing would show you:

● CAGR: 12.01% a year (11.72% after a 0.05% cost on each buy and sell)

● Trades: 27, and 63% of them made money

● Average win vs average loss: +8.1% vs -2.4%

● Sharpe ratio: 1.10, vs 0.75 for the Nifty

● Worst drawdown: -14.2%, vs -38.4% for the Nifty

That drawdown number is the sales pitch. Same market, a third of the pain. But look one line up: the strategy actually made less than just buying the Nifty and holding it (12.01% vs 12.65%). People would still buy it, because what's really being sold is the smooth ride. So the real question is: is that smoothness skill, or luck?

3. Is the edge real? The test that matters gives p = 0.88

Compared to just holding the Nifty, this strategy is statistically no different.

The biggest thing Module 3 taught me is that a backtest is a sample. Those 2,464 days are one possible version of history. A different decade would give different numbers. So the question isn't "what was the average return?" It's "how confident can I be that the real average isn't zero, or isn't just the market?" That's exactly what a 95% confidence interval and a t-test are for.

I ran four tests:

And here's what came out:

Test A: strategy vs zero. Looks amazing. Average daily return is +0.046%, the whole confidence interval is above zero, and p = 0.001. If I stopped here, I could say "statistically significant!" and a lot of people would believe me. Test B: Nifty vs zero. This is why I didn't stop. The Nifty on its own also passes the test (p = 0.020). Indian stocks went up from 2016 to 2025. So beating zero just means you were invested for some of that time. Zero was the wrong thing to compare against. The real "do nothing" option isn't cash. It's just holding the index.

Test C: strategy vs Nifty. This is the test that actually matters. For each of the 2,464 days, I took the strategy's return minus the Nifty's return, then tested whether that difference averages out to zero. It has to be a paired test, because both returns come from the same days and move together. The average difference is -0.002% a day, the confidence interval goes from -0.033% to +0.029%, and p = 0.884. Zero sits right in the middle.

Test D: per trade. 27 trades, averaging +4.2% each, p = 0.007. Sounds great, but it's the same trap as Test A. Each "trade" is just a period of holding the Nifty while the Nifty was going up.

So the strategy didn't make more money than the index. What it did do was take less risk. That's a real claim, and testing it needs a different tool.

4. Real skill, or one lucky quarter in 2020?

The regression found 6.5% a year of "alpha" that was statistically significant. Then I removed fifteen weeks of 2020, and it vanished.

Module 4 is all about regression. The cleanest way to ask "did this beat the market for the risk it took?" is to regress the strategy's daily returns on the Nifty's daily returns. You get two numbers:

● Beta: how much of the market's movement the strategy carries.

● Alpha: the extra return that the market can't explain. This is the "skill" part.

Beta came out at 0.42, so the strategy moves about 42% as much as the Nifty. Alpha came out at +6.5% a year, with p = 0.012. Honestly, this is where I got excited. A significant alpha is the kind of thing people make whole YouTube videos about.

So I tried to break it.

Test 1: remove one event. I deleted just 15 February to 31 May 2020 from the data. Alpha dropped to 2.4% a year, with p = 0.28. Not significant anymore. In those fifteen weeks, the strategy jumped about 33% ahead of the Nifty, because it happened to switch to cash right before the Covid crash. Over the other 9.7 years, it fell about 29% behind.

This chart shows it best. The red band is those fifteen weeks:

Test 2: split the decade in half. From 2016 to 2020 (which includes the crash), alpha was 12.0% a year, p = 0.002. From 2021 to 2025, it was -0.15% a year, p = 0.96. Basically zero.

Test 3: try other settings. Here's what people actually do with unlimited free backtests: they try lots of settings and keep the best one. So I did the same. I tested 23 different moving average pairs, from 5/50 up to 50/200. Five of them showed "significant" alpha.

But if you run 23 tests, you'd expect about one to look significant just by luck. So you have to make the bar higher. With a Bonferroni correction (divide 0.05 by 23), none of them pass.

It gets worse. I picked the five best pairs using only 2016-2020 data, then ran them on 2021-2025. Their alpha went from 11.5% to 14.4% a year down to roughly zero. Across all 23 pairs, the correlation between "good in the past" and "good in the future" was -0.02. Past winners told you nothing.

Was it just global markets in disguise? No.

My original guess for this post was that the "edge" would really just be the Nifty copying overnight moves in the US market, oil and the rupee. So I tested that too. Since US markets, Brent oil and the rupee rate all close after the NSE does, I matched each one to the next day in India.

The S&P 500's overnight move does matter (coefficient 0.12, p < 0.001). Oil and the rupee don't. But all three together explain just 4% of the strategy's daily moves, and only 7.9% of the Nifty's own. The strategy actually has less global exposure than the Nifty, because it's in cash 31% of the time. So my guess was wrong. I'm leaving that in, because a post about not trusting your first result shouldn't hide its own.

The real answer is simpler: the strategy is just a less risky version of the Nifty that got lucky and dodged one crash.

5. Why a smooth backtest can hide a disaster

Most risk numbers assume returns follow a nice bell curve. Nifty returns really don't.

Module 2 of the course teaches you to look at the actual shape of the returns, not just the average and standard deviation. For the Nifty from 2016 to 2025:

● Skew is -1.40. The big moves are mostly down moves. A normal bell curve would be 0.

● Excess kurtosis is 22.1. Extreme days happen way more than a bell curve predicts.

Again, normal would be 0.

● Worst day: 23 March 2020, a 13.9% drop (in log terms).

● A Jarque-Bera test, which checks if data is normal, gives a p-value of basically zero.

On days the strategy was invested, there were 18 days with moves bigger than 3 standard deviations. A normal bell curve says you'd expect about 4.6. The strategy looks smooth partly because it dodged some of the worst days: its worst day was -6.1%, while the Nifty's was -13.9%. That's literally the job of a trend-following strategy. Get out when things start breaking.

Now flip it around. The 9:20 straddle is one of the most popular automated options strategies in India. Every morning at 9:20, it sells a call and a put and collects the premium. Most days, it makes a little money. But on a big gap or crash day, it loses a lot. It's basically betting on exactly the kind of day the crossover avoided. So if your backtest period doesn't include a March 2020, it looks like free money: high Sharpe, high win rate. I couldn't backtest it myself because the intraday options data isn't free, but you can see the risk right there in the chart.

And this is who these tools are being sold to. SEBI's own study found that about 91% of individual F&O traders lost money in FY25. Net losses rose 41% to ₹1,05,603 crore, across roughly 96 lakh traders. To be fair, that's all F&O traders, not just straddle sellers. But the gap between a smooth-looking backtest and that result is hiding in the tails, not the average.

6. Jane Street: what a real edge looks like

The Jane Street case is a fight over whether a real edge was used unfairly. Retail algo trading has the opposite problem: edges that were never real.

On 3 July 2025, SEBI banned Jane Street from Indian markets and impounded ₹4,843.57 crore. SEBI alleged a two-part pattern: heavy buying of Bank Nifty stocks to push the index up, which supposedly helped a much bigger options position. Jane Street paid the amount and the ban was lifted. In September 2025 it appealed to the Securities Appellate Tribunal, saying it hadn't been given documents it needed to defend itself, and that an earlier SEBI inspection had found no manipulation. The latest update I could find is SAT postponing a hearing in February 2026. As of writing this, it's still unresolved.

Whoever wins, it's a useful comparison. SEBI's new rules (strategy IDs, broker responsibility, traceable orders) are built to catch the first kind of problem: someone with a real edge using it where they shouldn't. They do nothing about the second kind: lakhs of people trusting an edge that a 20-line t-test would have shown was luck.

It's the same idea as my Zepto DCF post. Take a standard tool, use it on something it wasn't built for, and see where it breaks.

7. "Obviously retail backtests aren't rigorous." Yeah, that's my point.

The problem isn't that free backtests aren't rigorous. It's that they're sold as if they are. The obvious pushback is: nobody expects a free backtest to be statistically perfect. Fair. But platforms show CAGR, win rate and drawdown to users as if those numbers are enough to make a decision. And most users have no idea what a confidence interval would have told them.

Everything in this post took me about 200 lines of Python and a free course. The fact that this strategy's whole decade of "edge" came from fifteen weeks in 2020 doesn't show up on any listing, anywhere.

If I could add two lines to every backtest result, they'd be:

1. The p-value of the strategy compared to just holding the index.

2. How much of the edge is left after you remove its single best quarter.

I think those two numbers would protect retail investors more than any strategy ID.

Methodology

● Data. Nifty 50 daily closes from a Yahoo Finance-derived series, cross-checked against a second independent copy with zero mismatches over 2,708 overlapping days. S&P 500 from FRED (SP500), Brent spot from the US EIA (RBRTE), USD/INR from FRED (DEXINUS).

● Returns. Price index, not total return. This slightly flatters the strategy, since buy-and-hold collects every dividend and the crossover only collects some.

● Costs. 0.05% per buy or sell for the net figure. No slippage or taxes. All tests use gross returns, the most generous version.

● Limits. T-tests assume roughly independent daily returns. Fat tails and volatility clustering bend that assumption, so Newey-West standard errors would be the stricter next step.

Sources

● SEBI algo framework extension, 30 Sep 2025

● SEBI algo timeline summary (Bajaj Broking)

● Algo-ID and broker responsibility (Sahi)

● Zerodha: Streak opened to all users, Jan 2024

● Zerodha FY26 active clients (Moneycontrol via TradingView)

● SEBI FY25 F&O study (Moneylife)

● Jane Street order details (Business Standard)

● Jane Street SAT appeal (PTI via The Week)

● SAT adjourns hearing, Feb 2026 (Business Today)

● Course: HKUST, Python and Statistics for Financial Analysis (Coursera), Modules 1 to 4