DeepLearning.AI

Josh Starmer

In this course, you will delve into the attention mechanism, a key component of transformers, and learn how to implement it using PyTorch. You'll explore the relationships between word embeddings, positional embeddings, and attention, and understand the roles of Query, Key, and Value matrices. The course covers self-attention, masked self-attention, and cross-attention, providing a comprehensive understanding of how these concepts are incorporated into transformers. By the end, you'll be equipped with the knowledge to build reliable and scalable AI applications.

Attention in Transformers: Concepts and Code in PyTorch

An overview of the course and its objectives.

Introduction

Exploration of the core concepts of transformers and the attention mechanism.

The Main Ideas Behind Transformers and Attention

Detailed explanation of the mathematical calculations involved in self-attention.

The Matrix Math for Calculating Self-Attention

Practical coding session to implement self-attention using PyTorch.

Coding Self-Attention in PyTorch

Comparison between self-attention and masked self-attention, highlighting their differences and uses.

Self-Attention vs Masked Self-Attention

Mathematical breakdown of masked self-attention calculations.

The Matrix Math for Calculating Masked Self-Attention

Hands-on coding session to implement masked self-attention in PyTorch.

Coding Masked Self-Attention in PyTorch

Understanding the role of attention in the encoder-decoder architecture.

Encoder-Decoder Attention

Exploration of multi-head attention and its significance in transformers.

Multi-Head Attention

Practical coding session to implement encoder-decoder and multi-head attention using PyTorch.

Coding Encoder-Decoder Attention and Multi-Head Attention in PyTorch

Conclusion

Assessment to test understanding of the course material.

Quiz

Additional resources and tips for further learning.

Appendix – Tips and Help

This course provides a clear explanation of the attention mechanism in transformers, a breakthrough architecture powering large language models like ChatGPT. Learn how to code attention mechanisms in PyTorch and improve your understanding of AI applications.

Individuals with basic Python knowledge interested in learning about the attention mechanism in LLMs.

Python Enthusiasts

Developers looking to understand the foundational architecture of transformers to build scalable AI applications.

AI Developers

Data scientists aiming to enhance their understanding of attention mechanisms in large language models.

Data Scientists

Artificial Intelligence

Business

Cloud Engineering

Cybersecurity

Data

Design

Devops

Finance

Green Skills

Languages

AI Foundations

AI for Business

AI Automation & Workflows

Agentic AI

AI Coding / Vibe Coding

Agile and Scrum Methodologies

Agile and Scrum Practices 

AI Automation & Orchestration

AI for Data

AI for Design

Access 3,000+ courses tailored to your team's needs, covering AI, Cloud, Cybersecurity, Marketing, Languages, and more.

Mydra's AI Career CoPilot guides you with personalized learning paths tailored to your career goals, education, and experience.

We're more than an upskilling platform — with Mydra, you gain access to:

Mydra makes skill development accessible for your team with flexible payment solutions to fit any budget. Explore our options below.

The Benefits of Choosing Mydra

By canceling, you’ll lose access to all course materials and benefits. If you still wish to proceed, you can confirm below.

Pay now, in full with your personal card.

Credit & Debit Card

Based on your location, these are the available payment options. Click the flag to switch your location.

Which country are you from?

FUNDAE helps you grow professionally with employer-supported training and financial aid, fostering a culture of learning and innovation.

We'll help you design, finance, and manage the ideal training plan for you, maximizing FUNDAE benefits hassle-free.

This programme is eligible for Fundae funding. Your employer may be able to pay for this and reclaim the cost.

You can pause your payments if your monthly income is below {formattedMinThreshold}.

The remaining {remainingPayments} payments scheduled each month based on your income x {percentage}

As an employee, you can reduce your personal taxable income through eligible education deductions, spread over {numberOfInstalments} monthly payments. This combines the tax benefits with more manageable monthly payments from your gross income. To qualify, your employer must offer support for 'Retribución Flexible' and approve your course.

Based on the Retribución Flexible tax savings calculator, you could save {savings} on income tax. This estimate is based on a minimum annual gross salary of {minSalary}

The Tax Savings benefit is received through payroll as a reduction in tax deductions.

As an employee, you can reduce your personal taxable income through eligible education deductions. These tax benefits can be combined with any payment method and may result in significant savings on your annual tax return. However, your employer must offer support for 'Retribución Flexible' and approve your course.

Looks like your {availableLearningBudget} budget is a bit short—you're missing {requiredRemainingAmount} for that {courseCost} course!

Please tell us the date your salary is normally paid to your account so we can schedule your FixPay collection - this will normally be a few days later, but we'll confirm the exact date with you.

We have emailed you a copy for your records. We are now finalising the details and your financial product details will appear here soon.

Thanks for signing the contract!

{classTypes} classes

Once you confirm your course selection, you will receive an email containing the course URL, which will take you directly to the {epName}'s page.

Access your tax benefits or use your employer learning budget to bundle with your payment method.

If you can't find the answer you're looking for, chat with your Mydra advisor!

Frequently Asked Questions

We are currently reviewing the validation of your tax benefits. We will contact you as soon as we have more information.

Split the payments in monthly instalments, or combine it with tax credits or learning budget. To learn more click

Pay your course in full, or combine your upfront payment with tax credits or your employee learning budget. To learn more, click

Don’t know what to learn? We’ll help you discover your next step for success.

Generate your AI-learning path

Learn on your terms. Filter courses based on your skill needs, budget and time.

Tailor your search

Find your course, pay smooth, learn like a pro

Courses Marketplace | Mydra

Name

Please tell us if you have ever received any support from the European Investment Fund

Brief description (tell us about your organization in a few words)

Please add the provider's authorized legal representative to handle updates to the team's legal contract. Without this contact information, the contract cannot be signed, which will delay the sale of the courses. Providing these details promptly will ensure a smooth and timely process.

Please complete this information if you would like to set up the provider's account now, or you can do it later.

A Legal Representative of an Education Provider is authorized to act on the institution's behalf in legal, administrative, and contractual matters. They ensure regulatory compliance, manage agreements, represent the provider in official matters, handle critical decisions, and serve as the primary contact for partnerships and negotiations.

Your request will be forwarded to our partners, and we will ensure the price is made available to you promptly. You will receive an email notification once the price is available. Thank you for your request. Feel free to explore other courses for more exciting learning opportunities here.

Would you like to request the price for this cohort?

Leave it with us - we'll be back in contact as soon as possible

Would you like to find out what dates this course is available?

We will analyze your request, and once these features are available, you will receive an email confirmation. If you want to know more about the benefits of these features, please visit our <Link>Payment Options</Link> page.

Would you like to activate the {paymentFeature, select, learningBudget {Learning Budget} taxBenefits {Tax Benefits} other {this}} feature for this course?

Many companies set aside funds for employee training—let’s check if yours can cover this course!Please provide the contact details of the person in your company who manages L&D budgets. This is usually someone in HR, People, or Learning & Development, but if you're unsure, you can add your manager instead.

<title>What Happens Next?</title> <ul><li>✅ We contact your employer to confirm if they can fund this course for you.</li><li>✅ Once approved, we handle the payment process directly between your employer and the education provider.</li><li>✅ We keep you updated throughout the process.</li><li>📢 Tip: It’s helpful to flag this internally so they know to expect our request—it could speed things up!</li></ul>

Use Your Employer’s Learning & Development Budget for This Course

We will analyze your request, and once this payment option is available, you will receive an email confirmation. If you want to know more about the benefits of this payment option, please visit our <Link>Payment Options</Link> page.

Would you like to request {paymentMethod, select, upfront {Upfront} fix_pay {FixPay} other {}} as your payment option for this course?

We will analyze your request, and once this payment option is available, you will receive an email confirmation.

Please note that your company’s validation is required to use FUNDAE as a payment option. If your company has available FUNDAE credits, they can be used to cover the cost of eligible courses, allowing you to upskill at no additional expense to you.

Would you like to request FUNDAE as your payment option?

Payment Options

Please list the total amount you spend each month on the following three items. If you share your bills with someone else, just put your share.

If you’re already using Retribución Flexible, please input how much you’ve spent this year per category:

While the course fee remains the same, you can use your tax credits to reduce your overall yearly expenses.

Income Share Agreements (ISAs) offer a flexible and affordable solution for professional development. Employees can defer payments until they earn a stable income, reducing financial pressure while advancing their skills. For companies, ISAs ensure a highly skilled workforce without upfront costs, aligning employee growth with business success.

<Chip>{blockedLearningBudget}</Chip> will be deducted once your request is accepted.

As the world evolves, so should you. Upskill to stay ahead.

Check the details of your learning budget.

Join the Mydra community today!

FUNDAE is a crucial resource for companies looking to invest in their employees’ professional growth while benefiting from financial support. It plays an essential role in fostering a culture of learning and innovation in Spain. <ul><li>Reduces the financial burden of employee training.</li><li>Promotes skill development tailored to company needs.</li><li>Enhances competitiveness by upskilling the workforce.</li><li>Increases access to continuous professional development.</li><li>Improves employability and career prospects.</li></ul>

Pay using FUNDAE benefits

Pay via Income Share Agreement

Enhance the value of your training investments. Empower your employees to reach their full potential while aligning professional development with your business goals. Mydra helps you allocate your learning budget efficiently, track progress, and achieve measurable results—ensuring every dollar spent drives growth and success.<ul><li>Take advantage of your company's learning budget</li><li>Combine it with tax benefits or FixPay for maximum savings.</li></ul>

Mydra is dedicated to making skill development accessible for all your employees, seamlessly fitting into any budget. With flexible payment solutions tailored to your financial needs, we maximize learning opportunities for your team. Click below to explore our payment options and their benefits.

At Mydra, we facilitate access to and management of FUNDAE training subsidies for employees, allowing companies to make the most of the funds allocated for professional development.<ul><li>As an organizing company, Mydra handle all administrative and regulatory processes, ensuring compliance with legal requirements, optimizing the use of available training credits, and reducing the operational burden for educational institutions.</li><li>Mydra improves the accessibility of training programs while also increasing demand from companies looking to maximize the return on their investment in workforce development.</li></ul>

We are a FUNDAE organizing company

Mydra offers accessible financing for everyone by covering tuition costs upfront, ensuring that financial barriers don’t prevent learners from pursuing their education. Students only begin making payments after they complete their course, allowing them to focus entirely on their studies. Additionally, Mydra's Income Share Agreements (ISAs) provide exceptional downside protection—payments are paused anytime a learner’s income falls below the minimum threshold of €1,333.33, offering peace of mind and financial security.

Income Share Agreements. Removing financial friction

As a user, you will maximize the value of your training. Reach your full potential while advancing your professional development. Mydra helps users efficiently allocate the learning budget provided by some companies to their employees, track progress, and achieve measurable results, ensuring that every euro invested drives growth and success.<ul><li>Take advantage of your company's learning budget</li><li>Combine it with tax benefits or FixPay for maximum savings.</li></ul>

FixPay lets employees spread their learning costs over time with 0% interest, making professional growth more manageable.How to apply a FixPay:<ul><li>Select a course and 'Pay in installments - FixPay' </li><li>Provide basic information (name, address, salary).</li><li>Identity verification and recent payslip upload.</li><li>Sign a financial agreement.</li></ul>

FixPay

Pay in 0% interest instalments with FixPay

FUNDAE

Pay only when you have an income with ISA

Learning budget

Leverage your Employer Learning Budget

Maximize your training investments with government grants and flexible compensation options. Employees can save up to 30% of their pre-tax salary, making learning more affordable and effective. Compatible with:<Logos><FundaeLogo></FundaeLogo><EdenredLogo></EdenredLogo><CobeeLogo></CobeeLogo><SodexoLogo></SodexoLogo><BetterflyLogo></BetterflyLogo><CoverflexLogo></CoverflexLogo></Logos>

Tax benefits

Access Tax Benefits in two clicks

Smart Payment Options, Zero Hassle

To start, let's fill out this information. It will only take a few seconds.

After sign-up, you’ll able to

Designed for everyone eager to upskill, with flexible payment options available.

Flexible payment options for your learning journey

Easy Smart Payment Solutions Built for you

Spread across 12 monthly payments. Income Flex Protection.

0% Interest Instalments

Accessible for free through your employer. We take care of the admin.

Fundae: Premium courses

Pay a fixed % of your income. Payment Pause Guarantee.

ISA: No up front payment

Seamlessly access your company's training budget.

Employer subsidy

Ensure you're getting the most value out of every euro spent.

Access tax discounts

Take a moment to double-check your personal details and finalize any incomplete steps.

We're reviewing your application, which typically takes just a few hours. Check your summary below—click and click in 'View my Orders' to view details. Track your progress anytime on your dashboard.

You can proceed with the submission of your application by clicking the button below.

Ready to Future-Proof your Career

Whether you’re preparing for your first job, aiming for a promotion, or considering a career change, Mydra’s AI Career CoPilot is here to guide you. This intelligent tool designs a personalized learning path based on your career objectives, education, and work experience.

Tailored Learning Paths for Your Career Goals

Join the top training providers on Mydra today

This helps us personalize your experience. This will take less than a minute.

Which statement best reflects your team's AI experience?

How does your team currently handle repetitive tasks?

What does your company do? Tell us about your products or services.

e.g. We provide financial solutions for students, including tuition financing and flexible repayment plans.

This helps us tailor your experience. It'll take less than a minute.

Which best describes your company?

Select everything that applies — the more you tell us, the more accurate your Work Map will be. Add anything specific in the field below.

Please indicate your team's AI model preference.

Select all that apply — this helps shape your workflows and learning path from the start.

Please indicate your team's level of technical expertise.

This helps us personalize your workflow and recommendations.

Please select your team's primary automation need:

Interested in exploring additional content?

Attention in Transformers: Concepts and Code in PyTorch

Overview

Who is this course for?

What will you learn?

Upcoming cohorts